==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-cfd7686d9-6dlt4 Namespace: xlou Priority: 0 Node: gke-xlou-cdm-default-pool-f05840a3-bqd0/10.142.15.205 Start Time: Sun, 28 May 2023 05:58:35 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=cfd7686d9 skaffold.dev/run-id=e335f689-c21c-43c2-b220-f5ec9c667d23 Annotations: Status: Running IP: 10.106.43.43 IPs: IP: 10.106.43.43 Controlled By: ReplicaSet/lodemon-cfd7686d9 Containers: lodemon: Container ID: containerd://5f9846a11f60c00f2f942747c1351b8783cd63d0f037e314d9d83385428905f7 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:84da775f610cb515832f400ea8e94b080f2d431af1140fc36a41223c7933e003 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sun, 28 May 2023 05:58:36 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wwvjv (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-wwvjv: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 09:25:29 INFO 09:25:29 INFO [loop_until]: kubectl --namespace=xlou top pods 09:25:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:25:29 INFO [loop_until]: OK (rc = 0) 09:25:29 DEBUG --- stdout --- 09:25:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 9m 5814Mi am-7849cf7bdb-vcl2f 9m 5771Mi ds-cts-0 8m 455Mi ds-cts-1 8m 409Mi ds-cts-2 9m 426Mi ds-idrepo-0 6188m 9895Mi ds-idrepo-1 15m 13723Mi ds-idrepo-2 13m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 9092m 4319Mi idm-58dc667486-h86fb 7785m 4013Mi lodemon-cfd7686d9-6dlt4 1m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3242m 2686Mi 09:25:29 DEBUG --- stderr --- 09:25:29 DEBUG 09:25:31 INFO 09:25:31 INFO [loop_until]: kubectl --namespace=xlou top node 09:25:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:25:31 INFO [loop_until]: OK (rc = 0) 09:25:31 DEBUG --- stdout --- 09:25:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 93m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 72m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 125m 0% 1540Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 80m 0% 6815Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 12742m 80% 5520Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 7999m 50% 5175Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 79m 0% 1349Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1091Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1100Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1074Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 61m 0% 13154Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6165m 38% 10431Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 64m 0% 14322Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3364m 21% 3635Mi 6% 09:25:31 DEBUG --- stderr --- 09:25:31 DEBUG 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "ds-idrepo-0" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "ds-idrepo-1" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG } 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG ] 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG } 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG } 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "4593119232" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "ds-idrepo-0" 09:26:23 DEBUG "pod": "ds-idrepo-0" 09:26:23 DEBUG }, 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "ds-idrepo-1" 09:26:23 DEBUG "pod": "ds-cts-1" 09:26:23 DEBUG }, 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:26:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:26:23 DEBUG }, 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:26:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:26:23 DEBUG }, 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 DEBUG { 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG "data": { 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG "result": [ 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:26:23 DEBUG { 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG "metric": { 09:26:23 DEBUG { 09:26:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG }, 09:26:23 DEBUG "data": { 09:26:23 DEBUG "value": [ 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG 09:26:23 DEBUG "result": [ 09:26:23 DEBUG 09:26:23 DEBUG { 09:26:23 DEBUG 09:26:23 DEBUG "metric": { 09:26:23 DEBUG [...] 09:26:23 DEBUG "instance": "10.142.15.199:9100" 09:26:23 DEBUG 09:26:23 DEBUG }, 09:26:23 DEBUG 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:26:23 DEBUG 09:26:23 DEBUG }, 09:26:23 DEBUG 09:26:23 DEBUG "value": [ 09:26:23 DEBUG [...] 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG 09:26:23 DEBUG "0.2" 09:26:23 DEBUG 09:26:23 DEBUG ] 09:26:23 DEBUG 09:26:23 DEBUG } 09:26:23 DEBUG "instance": "10.142.15.212:9100" 09:26:23 DEBUG ] 09:26:23 DEBUG }, 09:26:23 DEBUG } 09:26:23 DEBUG "value": [ 09:26:23 DEBUG } 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0.00033333333333397756" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "ds-cts-2" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "ds-cts-1" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "instance": "10.142.15.199:9100" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "instance": "10.142.15.212:9100" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "31317.699999999997" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "instance": "10.142.15.199:9100" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "instance": "10.142.15.212:9100" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0.002766666666667087" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "instance": "10.142.15.199:9100" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "instance": "10.142.15.212:9100" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "28402.033333333333" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "ds-idrepo-0" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "ds-cts-1" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "ds-idrepo-0" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "ds-idrepo-1" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "3.222252448642971" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262383" 09:26:23 INFO [http_cmd]: http status code OK 09:26:23 DEBUG --- status code --- 09:26:23 DEBUG http status code is 200 (expected 200) 09:26:23 DEBUG --- http response --- 09:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:26:23 DEBUG ----- output ----- 09:26:23 DEBUG { 09:26:23 DEBUG "status": "success", 09:26:23 DEBUG "data": { 09:26:23 DEBUG "resultType": "vector", 09:26:23 DEBUG "result": [ 09:26:23 DEBUG { 09:26:23 DEBUG "metric": { 09:26:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG [...] 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG 09:26:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:26:23 DEBUG }, 09:26:23 DEBUG "value": [ 09:26:23 DEBUG 1685262383, 09:26:23 DEBUG "0" 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG ] 09:26:23 DEBUG } 09:26:23 DEBUG } 09:26:29 INFO 09:26:29 INFO [loop_until]: kubectl --namespace=xlou top pods 09:26:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:26:29 INFO [loop_until]: OK (rc = 0) 09:26:29 DEBUG --- stdout --- 09:26:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 9m 5814Mi am-7849cf7bdb-vcl2f 9m 5771Mi ds-cts-0 9m 455Mi ds-cts-1 8m 409Mi ds-cts-2 9m 426Mi ds-idrepo-0 6162m 9896Mi ds-idrepo-1 15m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 8855m 4325Mi idm-58dc667486-h86fb 7986m 4018Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3350m 2687Mi 09:26:29 DEBUG --- stderr --- 09:26:29 DEBUG 09:26:31 INFO 09:26:31 INFO [loop_until]: kubectl --namespace=xlou top node 09:26:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:26:31 INFO [loop_until]: OK (rc = 0) 09:26:31 DEBUG --- stdout --- 09:26:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 93m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 72m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 128m 0% 1533Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 77m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 12478m 78% 5526Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 7890m 49% 5172Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 77m 0% 1346Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1089Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 60m 0% 1100Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1073Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13158Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6246m 39% 10432Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14324Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3354m 21% 3633Mi 6% 09:26:31 DEBUG --- stderr --- 09:26:31 DEBUG 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "ds-idrepo-0" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "ds-idrepo-1" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262443" 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "4595179520" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "ds-idrepo-0" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "ds-idrepo-1" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "ds-idrepo-0" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "ds-cts-1" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262443" 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0.2" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262443" 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "instance": "10.142.15.199:9100" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "instance": "10.142.15.212:9100" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:27:23 DEBUG { 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG { 09:27:23 DEBUG "data": { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "result": [ 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG { 09:27:23 DEBUG "result": [ 09:27:23 DEBUG "metric": { 09:27:23 DEBUG { 09:27:23 DEBUG "pod": "ds-cts-2" 09:27:23 DEBUG "metric": { 09:27:23 DEBUG }, 09:27:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:27:23 DEBUG "value": [ 09:27:23 DEBUG }, 09:27:23 DEBUG 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "ds-cts-1" 09:27:23 DEBUG 09:27:23 DEBUG }, 09:27:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:27:23 DEBUG "value": [ 09:27:23 DEBUG }, 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG "0" 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG ] 09:27:23 DEBUG "0" 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "instance": "10.142.15.199:9100" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "instance": "10.142.15.212:9100" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "31343.266666666666" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262443" 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "instance": "10.142.15.199:9100" 09:27:23 DEBUG "instance": "10.142.15.199:9100" 09:27:23 DEBUG }, 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "instance": "10.142.15.212:9100" 09:27:23 DEBUG "instance": "10.142.15.212:9100" 09:27:23 DEBUG }, 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0.003200000000000121" 09:27:23 DEBUG "28592.833333333332" 09:27:23 DEBUG ] 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "ds-idrepo-0" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "ds-cts-1" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "ds-idrepo-0" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "ds-idrepo-1" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "3.2654957953940693" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262443" 09:27:23 INFO [http_cmd]: http status code OK 09:27:23 DEBUG --- status code --- 09:27:23 DEBUG http status code is 200 (expected 200) 09:27:23 DEBUG --- http response --- 09:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:27:23 DEBUG ----- output ----- 09:27:23 DEBUG { 09:27:23 DEBUG "status": "success", 09:27:23 DEBUG "data": { 09:27:23 DEBUG "resultType": "vector", 09:27:23 DEBUG "result": [ 09:27:23 DEBUG { 09:27:23 DEBUG "metric": { 09:27:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG [...] 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG 09:27:23 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:27:23 DEBUG }, 09:27:23 DEBUG "value": [ 09:27:23 DEBUG 1685262443, 09:27:23 DEBUG "0" 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG ] 09:27:23 DEBUG } 09:27:23 DEBUG } 09:27:29 INFO 09:27:29 INFO [loop_until]: kubectl --namespace=xlou top pods 09:27:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:27:29 INFO [loop_until]: OK (rc = 0) 09:27:29 DEBUG --- stdout --- 09:27:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 9m 5814Mi am-7849cf7bdb-vcl2f 9m 5771Mi ds-cts-0 9m 455Mi ds-cts-1 8m 409Mi ds-cts-2 8m 426Mi ds-idrepo-0 5924m 9896Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 8808m 4330Mi idm-58dc667486-h86fb 7965m 4025Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3159m 2688Mi 09:27:29 DEBUG --- stderr --- 09:27:29 DEBUG 09:27:31 INFO 09:27:31 INFO [loop_until]: kubectl --namespace=xlou top node 09:27:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:27:31 INFO [loop_until]: OK (rc = 0) 09:27:31 DEBUG --- stdout --- 09:27:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 91m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6659Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 131m 0% 1534Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 94m 0% 6821Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 12511m 78% 5530Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 8070m 50% 5175Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 81m 0% 1351Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1091Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1101Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1072Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 65m 0% 13156Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6119m 38% 10431Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 64m 0% 14323Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3286m 20% 3633Mi 6% 09:27:31 DEBUG --- stderr --- 09:27:31 DEBUG 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG { 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:28:23 DEBUG "data": { 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "pod": "ds-idrepo-0" 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG }, 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "value": [ 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:28:23 DEBUG 09:28:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:28:23 DEBUG }, 09:28:23 DEBUG 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG [...] 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG "pod": "ds-idrepo-1" 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG }, 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG 09:28:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:28:23 DEBUG "0" 09:28:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:28:23 DEBUG }, 09:28:23 DEBUG ] 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG } 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG ] 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "0" 09:28:23 DEBUG } 09:28:23 DEBUG "4600299520" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "pod": "ds-idrepo-0" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "pod": "ds-idrepo-1" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "0" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "0" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "pod": "ds-idrepo-0" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "pod": "ds-cts-1" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "0" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "data": { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:28:23 DEBUG }, 09:28:23 DEBUG "result": [ 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG { 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG "metric": { 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "instance": "10.142.15.199:9100" 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG }, 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG "value": [ 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG [...] 09:28:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:28:23 DEBUG }, 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "0.2" 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:28:23 DEBUG "instance": "10.142.15.212:9100" 09:28:23 DEBUG "0" 09:28:23 DEBUG ] 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG }, 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG { 09:28:23 DEBUG "value": [ 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG "data": { 09:28:23 DEBUG "0" 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG "result": [ 09:28:23 DEBUG } 09:28:23 DEBUG { 09:28:23 DEBUG ] 09:28:23 DEBUG "metric": { 09:28:23 DEBUG } 09:28:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:28:23 DEBUG } 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "49048.63849765259" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "4.175631691106584" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG { 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG "data": { 09:28:23 DEBUG { 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG "result": [ 09:28:23 DEBUG "data": { 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "result": [ 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:28:23 DEBUG "pod": "ds-cts-2" 09:28:23 DEBUG { 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG }, 09:28:23 DEBUG "metric": { 09:28:23 DEBUG { 09:28:23 DEBUG "value": [ 09:28:23 DEBUG "instance": "10.142.15.199:9100" 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG 09:28:23 DEBUG }, 09:28:23 DEBUG "data": { 09:28:23 DEBUG 09:28:23 DEBUG "value": [ 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "result": [ 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG { 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262503" 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "metric": { 09:28:23 DEBUG 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 DEBUG [...] 09:28:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG }, 09:28:23 DEBUG "pod": "ds-cts-1" 09:28:23 DEBUG 09:28:23 DEBUG "value": [ 09:28:23 DEBUG }, 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "value": [ 09:28:23 DEBUG "instance": "10.142.15.212:9100" 09:28:23 DEBUG 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG }, 09:28:23 DEBUG 09:28:23 DEBUG "0" 09:28:23 DEBUG "value": [ 09:28:23 DEBUG [...] 09:28:23 DEBUG ] 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG 09:28:23 DEBUG } 09:28:23 DEBUG "31397.93333333333" 09:28:23 DEBUG 09:28:23 DEBUG ] 09:28:23 DEBUG ] 09:28:23 DEBUG 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG }, 09:28:23 DEBUG } 09:28:23 DEBUG "value": [ 09:28:23 DEBUG } 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "0" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "instance": "10.142.15.199:9100" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "instance": "10.142.15.212:9100" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "29989.199999999997" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "0" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262503" 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "instance": "10.142.15.199:9100" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "instance": "10.142.15.212:9100" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "0.0025333333333331133" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "result": [ 09:28:23 DEBUG { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "pod": "ds-idrepo-0" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "pod": "ds-cts-1" 09:28:23 DEBUG }, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "0" 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262503" 09:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262503" 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG --- status code --- 09:28:23 INFO [http_cmd]: http status code OK 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG --- status code --- 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG http status code is 200 (expected 200) 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:28:23 DEBUG --- http response --- 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:28:23 DEBUG { 09:28:23 DEBUG { 09:28:23 DEBUG ----- output ----- 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG { 09:28:23 DEBUG "data": { 09:28:23 DEBUG "data": { 09:28:23 DEBUG "status": "success", 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG "data": { 09:28:23 DEBUG "result": [ 09:28:23 DEBUG "result": [ 09:28:23 DEBUG "resultType": "vector", 09:28:23 DEBUG { 09:28:23 DEBUG { 09:28:23 DEBUG "result": [ 09:28:23 DEBUG "metric": { 09:28:23 DEBUG "metric": { 09:28:23 DEBUG { 09:28:23 DEBUG "pod": "ds-idrepo-0" 09:28:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:28:23 DEBUG "metric": { 09:28:23 DEBUG }, 09:28:23 DEBUG }, 09:28:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:28:23 DEBUG "value": [ 09:28:23 DEBUG "value": [ 09:28:23 DEBUG }, 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "value": [ 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG [...] 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG 09:28:23 DEBUG "pod": "ds-idrepo-1" 09:28:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:28:23 DEBUG 09:28:23 DEBUG }, 09:28:23 DEBUG }, 09:28:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:28:23 DEBUG "value": [ 09:28:23 DEBUG "value": [ 09:28:23 DEBUG }, 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG "value": [ 09:28:23 DEBUG "0" 09:28:23 DEBUG "0" 09:28:23 DEBUG 1685262503, 09:28:23 DEBUG ] 09:28:23 DEBUG ] 09:28:23 DEBUG "3.2403388569501277" 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG ] 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG ] 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:23 DEBUG } 09:28:29 INFO 09:28:29 INFO [loop_until]: kubectl --namespace=xlou top pods 09:28:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:28:29 INFO [loop_until]: OK (rc = 0) 09:28:29 DEBUG --- stdout --- 09:28:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 6m 5743Mi am-7849cf7bdb-cwb58 9m 5814Mi am-7849cf7bdb-vcl2f 9m 5771Mi ds-cts-0 8m 455Mi ds-cts-1 9m 409Mi ds-cts-2 9m 426Mi ds-idrepo-0 5886m 9895Mi ds-idrepo-1 15m 13723Mi ds-idrepo-2 16m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 8707m 4338Mi idm-58dc667486-h86fb 7595m 4029Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3153m 2688Mi 09:28:29 DEBUG --- stderr --- 09:28:29 DEBUG 09:28:31 INFO 09:28:31 INFO [loop_until]: kubectl --namespace=xlou top node 09:28:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:28:31 INFO [loop_until]: OK (rc = 0) 09:28:31 DEBUG --- stdout --- 09:28:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 89m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6661Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 143m 0% 1537Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 12652m 79% 5549Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 7886m 49% 5179Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1091Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1102Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1072Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13158Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 5979m 37% 10430Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 64m 0% 14325Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3308m 20% 3632Mi 6% 09:28:31 DEBUG --- stderr --- 09:28:31 DEBUG 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "ds-idrepo-2" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0.9006796037009743" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "ds-idrepo-0" 09:29:23 DEBUG "pod": "ds-idrepo-0" 09:29:23 DEBUG }, 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "ds-idrepo-1" 09:29:23 DEBUG "pod": "ds-idrepo-1" 09:29:23 DEBUG }, 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "4605992960" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "ds-idrepo-0" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "ds-cts-1" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262563" 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "instance": "10.142.15.199:9100" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "instance": "10.142.15.212:9100" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG { 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:29:23 DEBUG "data": { 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG { 09:29:23 DEBUG "result": [ 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG { 09:29:23 DEBUG "data": { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:29:23 DEBUG "result": [ 09:29:23 DEBUG }, 09:29:23 DEBUG { 09:29:23 DEBUG "value": [ 09:29:23 DEBUG "metric": { 09:29:23 DEBUG 09:29:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:29:23 DEBUG 09:29:23 DEBUG }, 09:29:23 DEBUG 09:29:23 DEBUG "value": [ 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG "pod": "ds-idrepo-2" 09:29:23 DEBUG 09:29:23 DEBUG }, 09:29:23 DEBUG 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:29:23 DEBUG "383217082.562297" 09:29:23 DEBUG }, 09:29:23 DEBUG ] 09:29:23 DEBUG "value": [ 09:29:23 DEBUG } 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG ] 09:29:23 DEBUG "0.2" 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262563" 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "instance": "10.142.15.199:9100" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "instance": "10.142.15.212:9100" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "31388.4" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "instance": "10.142.15.199:9100" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "instance": "10.142.15.212:9100" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "28906.833333333332" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "ds-cts-2" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "ds-cts-1" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "ds-idrepo-2" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0.01401270296678318" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "instance": "10.142.15.199:9100" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "instance": "10.142.15.212:9100" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0.0028666666666661436" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "ds-idrepo-0" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "ds-cts-1" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "result": [ 09:29:23 DEBUG { 09:29:23 DEBUG "metric": { 09:29:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "ds-idrepo-2" 09:29:23 DEBUG }, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "0" 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262563" 09:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262563" 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG --- status code --- 09:29:23 INFO [http_cmd]: http status code OK 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG --- status code --- 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG http status code is 200 (expected 200) 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:29:23 DEBUG --- http response --- 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:29:23 DEBUG { 09:29:23 DEBUG ----- output ----- 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG { 09:29:23 DEBUG "data": { 09:29:23 DEBUG "status": "success", 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG "data": { 09:29:23 DEBUG "result": [ 09:29:23 DEBUG "resultType": "vector", 09:29:23 DEBUG { 09:29:23 DEBUG "result": [ 09:29:23 DEBUG "metric": { 09:29:23 DEBUG { 09:29:23 DEBUG "pod": "ds-idrepo-0" 09:29:23 DEBUG "metric": { 09:29:23 DEBUG }, 09:29:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:29:23 DEBUG "value": [ 09:29:23 DEBUG }, 09:29:23 DEBUG 09:29:23 DEBUG "value": [ 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG [...] 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG 09:29:23 DEBUG "pod": "ds-idrepo-1" 09:29:23 DEBUG 09:29:23 DEBUG }, 09:29:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:29:23 DEBUG "value": [ 09:29:23 DEBUG }, 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG "value": [ 09:29:23 DEBUG "0" 09:29:23 DEBUG 1685262563, 09:29:23 DEBUG ] 09:29:23 DEBUG "3.246823396726899" 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 DEBUG ] 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:23 DEBUG } 09:29:29 INFO 09:29:29 INFO [loop_until]: kubectl --namespace=xlou top pods 09:29:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:29:29 INFO [loop_until]: OK (rc = 0) 09:29:29 DEBUG --- stdout --- 09:29:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 6m 5742Mi am-7849cf7bdb-cwb58 10m 5814Mi am-7849cf7bdb-vcl2f 9m 5771Mi ds-cts-0 8m 456Mi ds-cts-1 8m 409Mi ds-cts-2 9m 426Mi ds-idrepo-0 6062m 9896Mi ds-idrepo-1 13m 13723Mi ds-idrepo-2 13m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 8656m 4344Mi idm-58dc667486-h86fb 7889m 4035Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3192m 2689Mi 09:29:29 DEBUG --- stderr --- 09:29:29 DEBUG 09:29:31 INFO 09:29:31 INFO [loop_until]: kubectl --namespace=xlou top node 09:29:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:29:31 INFO [loop_until]: OK (rc = 0) 09:29:31 DEBUG --- stdout --- 09:29:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 85m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 69m 0% 6659Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 134m 0% 1532Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6815Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 12256m 77% 5540Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 7894m 49% 5185Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 80m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1091Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1102Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1072Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 61m 0% 13159Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6193m 38% 10427Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14322Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3297m 20% 3635Mi 6% 09:29:31 DEBUG --- stderr --- 09:29:31 DEBUG 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262623" 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "4610293760" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "ds-idrepo-0" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "ds-idrepo-1" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "ds-idrepo-0" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "ds-idrepo-1" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "instance": "10.142.15.199:9100" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "instance": "10.142.15.212:9100" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0.00033333333333397756" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:30:23 DEBUG "pod": "ds-idrepo-0" 09:30:23 DEBUG }, 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:30:23 DEBUG "pod": "ds-cts-1" 09:30:23 DEBUG }, 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0.2" 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "instance": "10.142.15.199:9100" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "instance": "10.142.15.212:9100" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "31596.36666666667" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "ds-idrepo-2" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "ds-cts-2" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "ds-cts-1" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "instance": "10.142.15.199:9100" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "instance": "10.142.15.212:9100" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "28883.8" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "ds-idrepo-2" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:30:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:30:23 DEBUG }, 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:30:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:30:23 DEBUG }, 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "instance": "10.142.15.199:9100" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "instance": "10.142.15.212:9100" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0.0026333333333328802" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:30:23 DEBUG { 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG { 09:30:23 DEBUG "data": { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "result": [ 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG { 09:30:23 DEBUG "result": [ 09:30:23 DEBUG "metric": { 09:30:23 DEBUG { 09:30:23 DEBUG "pod": "ds-idrepo-0" 09:30:23 DEBUG "metric": { 09:30:23 DEBUG }, 09:30:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:30:23 DEBUG "value": [ 09:30:23 DEBUG }, 09:30:23 DEBUG 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "ds-cts-1" 09:30:23 DEBUG 09:30:23 DEBUG }, 09:30:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:30:23 DEBUG "value": [ 09:30:23 DEBUG }, 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG "0" 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG ] 09:30:23 DEBUG "0" 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262623" 09:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262623" 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "pod": "ds-idrepo-0" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "pod": "ds-idrepo-1" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "0" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:23 INFO [http_cmd]: http status code OK 09:30:23 DEBUG --- status code --- 09:30:23 DEBUG http status code is 200 (expected 200) 09:30:23 DEBUG --- http response --- 09:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:30:23 DEBUG ----- output ----- 09:30:23 DEBUG { 09:30:23 DEBUG "status": "success", 09:30:23 DEBUG "data": { 09:30:23 DEBUG "resultType": "vector", 09:30:23 DEBUG "result": [ 09:30:23 DEBUG { 09:30:23 DEBUG "metric": { 09:30:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG [...] 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG 09:30:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:30:23 DEBUG }, 09:30:23 DEBUG "value": [ 09:30:23 DEBUG 1685262623, 09:30:23 DEBUG "3.2314398620467353" 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG ] 09:30:23 DEBUG } 09:30:23 DEBUG } 09:30:29 INFO 09:30:29 INFO [loop_until]: kubectl --namespace=xlou top pods 09:30:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:30:29 INFO [loop_until]: OK (rc = 0) 09:30:29 DEBUG --- stdout --- 09:30:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 10m 5814Mi am-7849cf7bdb-vcl2f 9m 5771Mi ds-cts-0 8m 455Mi ds-cts-1 9m 409Mi ds-cts-2 8m 426Mi ds-idrepo-0 5874m 9896Mi ds-idrepo-1 15m 13723Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 8704m 4349Mi idm-58dc667486-h86fb 7587m 4042Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3179m 2689Mi 09:30:29 DEBUG --- stderr --- 09:30:29 DEBUG 09:30:31 INFO 09:30:31 INFO [loop_until]: kubectl --namespace=xlou top node 09:30:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:30:31 INFO [loop_until]: OK (rc = 0) 09:30:31 DEBUG --- stdout --- 09:30:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 88m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 67m 0% 6659Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 131m 0% 1535Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6813Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 12416m 78% 5542Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 7894m 49% 5188Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 77m 0% 1349Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1089Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1101Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 57m 0% 1073Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13157Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6033m 37% 10430Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14321Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3239m 20% 3628Mi 6% 09:30:31 DEBUG --- stderr --- 09:30:31 DEBUG 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:31:23 DEBUG { 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG { 09:31:23 DEBUG "data": { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "result": [ 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG { 09:31:23 DEBUG "result": [ 09:31:23 DEBUG "metric": { 09:31:23 DEBUG { 09:31:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:31:23 DEBUG "metric": { 09:31:23 DEBUG }, 09:31:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:31:23 DEBUG "value": [ 09:31:23 DEBUG }, 09:31:23 DEBUG 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:31:23 DEBUG 09:31:23 DEBUG }, 09:31:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:31:23 DEBUG "value": [ 09:31:23 DEBUG }, 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG "0" 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG ] 09:31:23 DEBUG "4616073216" 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "ds-idrepo-0" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "ds-idrepo-1" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "ds-idrepo-0" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "ds-idrepo-1" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "ds-idrepo-0" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "ds-cts-1" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0.03333333333333333" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "instance": "10.142.15.199:9100" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "instance": "10.142.15.212:9100" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "31251.833333333332" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "instance": "10.142.15.199:9100" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "instance": "10.142.15.212:9100" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0.2" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "88535.11743772241" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "ds-cts-2" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "ds-cts-1" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "instance": "10.142.15.199:9100" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "instance": "10.142.15.212:9100" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "28337.166666666664" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:31:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:31:23 DEBUG }, 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:31:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:31:23 DEBUG }, 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0" 09:31:23 DEBUG "6.344011566084372" 09:31:23 DEBUG ] 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "instance": "10.142.15.199:9100" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "instance": "10.142.15.212:9100" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0.0028666666666666172" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG { 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG "data": { 09:31:23 DEBUG { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG "data": { 09:31:23 DEBUG { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "result": [ 09:31:23 DEBUG "pod": "ds-idrepo-0" 09:31:23 DEBUG { 09:31:23 DEBUG }, 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "value": [ 09:31:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:31:23 DEBUG 09:31:23 DEBUG }, 09:31:23 DEBUG 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "ds-cts-1" 09:31:23 DEBUG 09:31:23 DEBUG }, 09:31:23 DEBUG 09:31:23 DEBUG "value": [ 09:31:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG }, 09:31:23 DEBUG "0" 09:31:23 DEBUG "value": [ 09:31:23 DEBUG ] 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG } 09:31:23 DEBUG "0" 09:31:23 DEBUG ] 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "pod": "ds-idrepo-0" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "pod": "ds-idrepo-1" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "0" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262683" 09:31:23 INFO [http_cmd]: http status code OK 09:31:23 DEBUG --- status code --- 09:31:23 DEBUG http status code is 200 (expected 200) 09:31:23 DEBUG --- http response --- 09:31:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:31:23 DEBUG ----- output ----- 09:31:23 DEBUG { 09:31:23 DEBUG "status": "success", 09:31:23 DEBUG "data": { 09:31:23 DEBUG "resultType": "vector", 09:31:23 DEBUG "result": [ 09:31:23 DEBUG { 09:31:23 DEBUG "metric": { 09:31:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG [...] 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG 09:31:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:31:23 DEBUG }, 09:31:23 DEBUG "value": [ 09:31:23 DEBUG 1685262683, 09:31:23 DEBUG "3.217090275356258" 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG ] 09:31:23 DEBUG } 09:31:23 DEBUG } 09:31:30 INFO 09:31:30 INFO [loop_until]: kubectl --namespace=xlou top pods 09:31:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:31:30 INFO [loop_until]: OK (rc = 0) 09:31:30 DEBUG --- stdout --- 09:31:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 10m 5814Mi am-7849cf7bdb-vcl2f 9m 5766Mi ds-cts-0 9m 455Mi ds-cts-1 8m 409Mi ds-cts-2 8m 426Mi ds-idrepo-0 5610m 9896Mi ds-idrepo-1 14m 13723Mi ds-idrepo-2 13m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 8787m 4356Mi idm-58dc667486-h86fb 7747m 4047Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3080m 2690Mi 09:31:30 DEBUG --- stderr --- 09:31:30 DEBUG 09:31:31 INFO 09:31:31 INFO [loop_until]: kubectl --namespace=xlou top node 09:31:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:31:31 INFO [loop_until]: OK (rc = 0) 09:31:31 DEBUG --- stdout --- 09:31:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 90m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 76m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 119m 0% 1537Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 87m 0% 6815Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 12315m 77% 5550Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 7858m 49% 5201Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 82m 0% 1346Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 58m 0% 1089Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 63m 0% 1102Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1074Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 65m 0% 13158Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 5998m 37% 10430Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14324Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3281m 20% 3634Mi 6% 09:31:31 DEBUG --- stderr --- 09:31:31 DEBUG 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "4629524480" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "ds-idrepo-0" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "ds-idrepo-1" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "ds-idrepo-0" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "ds-idrepo-1" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262743" 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "instance": "10.142.15.199:9100" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "instance": "10.142.15.212:9100" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0.0003333333333330302" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "instance": "10.142.15.199:9100" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "instance": "10.142.15.212:9100" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "31263.533333333333" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "ds-idrepo-0" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "ds-cts-1" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0.2" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "373689.0144350518" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "ds-cts-2" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "ds-cts-1" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "instance": "10.142.15.199:9100" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "instance": "10.142.15.212:9100" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "28400.166666666664" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG { 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG { 09:32:23 DEBUG "data": { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "result": [ 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG { 09:32:23 DEBUG "result": [ 09:32:23 DEBUG "metric": { 09:32:23 DEBUG { 09:32:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:32:23 DEBUG "metric": { 09:32:23 DEBUG }, 09:32:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:32:23 DEBUG "value": [ 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG }, 09:32:23 DEBUG 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG { 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG 09:32:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:32:23 DEBUG "data": { 09:32:23 DEBUG 09:32:23 DEBUG }, 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:32:23 DEBUG "value": [ 09:32:23 DEBUG "result": [ 09:32:23 DEBUG }, 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG { 09:32:23 DEBUG "value": [ 09:32:23 DEBUG "0" 09:32:23 DEBUG "metric": { 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG ] 09:32:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:32:23 DEBUG "6.317796299374994" 09:32:23 DEBUG } 09:32:23 DEBUG }, 09:32:23 DEBUG ] 09:32:23 DEBUG ] 09:32:23 DEBUG "value": [ 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 DEBUG 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG 09:32:23 DEBUG } 09:32:23 DEBUG 09:32:23 DEBUG } 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "instance": "10.142.15.199:9100" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "instance": "10.142.15.212:9100" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0.002800000000000343" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "ds-idrepo-0" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "ds-cts-1" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "pod": "ds-idrepo-0" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "pod": "ds-idrepo-1" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "0" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262743" 09:32:23 INFO [http_cmd]: http status code OK 09:32:23 DEBUG --- status code --- 09:32:23 DEBUG http status code is 200 (expected 200) 09:32:23 DEBUG --- http response --- 09:32:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:32:23 DEBUG ----- output ----- 09:32:23 DEBUG { 09:32:23 DEBUG "status": "success", 09:32:23 DEBUG "data": { 09:32:23 DEBUG "resultType": "vector", 09:32:23 DEBUG "result": [ 09:32:23 DEBUG { 09:32:23 DEBUG "metric": { 09:32:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG [...] 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG 09:32:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:32:23 DEBUG }, 09:32:23 DEBUG "value": [ 09:32:23 DEBUG 1685262743, 09:32:23 DEBUG "3.163205797920571" 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG ] 09:32:23 DEBUG } 09:32:23 DEBUG } 09:32:30 INFO 09:32:30 INFO [loop_until]: kubectl --namespace=xlou top pods 09:32:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:32:30 INFO [loop_until]: OK (rc = 0) 09:32:30 DEBUG --- stdout --- 09:32:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 10m 5814Mi am-7849cf7bdb-vcl2f 9m 5766Mi ds-cts-0 9m 455Mi ds-cts-1 9m 409Mi ds-cts-2 9m 426Mi ds-idrepo-0 6065m 9896Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 13m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 8450m 4361Mi idm-58dc667486-h86fb 7770m 4060Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3093m 2690Mi 09:32:30 DEBUG --- stderr --- 09:32:30 DEBUG 09:32:31 INFO 09:32:31 INFO [loop_until]: kubectl --namespace=xlou top node 09:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:32:32 INFO [loop_until]: OK (rc = 0) 09:32:32 DEBUG --- stdout --- 09:32:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 87m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 76m 0% 6651Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 129m 0% 1535Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 84m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 12025m 75% 5555Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 7912m 49% 5206Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 82m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1095Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 62m 0% 1098Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1071Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13157Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6069m 38% 10431Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14324Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3290m 20% 3636Mi 6% 09:32:32 DEBUG --- stderr --- 09:32:32 DEBUG 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "ds-cts-0" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "4637544448" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "ds-idrepo-0" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "ds-idrepo-1" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "ds-idrepo-0" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "ds-idrepo-1" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262803" 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262803" 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "instance": "10.142.15.199:9100" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "instance": "10.142.15.212:9100" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG { 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:33:23 DEBUG "data": { 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG { 09:33:23 DEBUG "result": [ 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG { 09:33:23 DEBUG "data": { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "pod": "ds-idrepo-0" 09:33:23 DEBUG "result": [ 09:33:23 DEBUG }, 09:33:23 DEBUG { 09:33:23 DEBUG "value": [ 09:33:23 DEBUG "metric": { 09:33:23 DEBUG 09:33:23 DEBUG "instance": "10.142.15.199:9100" 09:33:23 DEBUG 09:33:23 DEBUG }, 09:33:23 DEBUG 09:33:23 DEBUG "value": [ 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG "pod": "ds-cts-1" 09:33:23 DEBUG 09:33:23 DEBUG }, 09:33:23 DEBUG 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "instance": "10.142.15.212:9100" 09:33:23 DEBUG "0" 09:33:23 DEBUG }, 09:33:23 DEBUG ] 09:33:23 DEBUG "value": [ 09:33:23 DEBUG } 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG ] 09:33:23 DEBUG "31524.7" 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0.2" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "ds-cts-0" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "13291.230635493885" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "ds-cts-2" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "ds-cts-1" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "instance": "10.142.15.199:9100" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "instance": "10.142.15.212:9100" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "30660.666666666664" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:33:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:33:23 DEBUG }, 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:33:23 DEBUG "pod": "ds-cts-0" 09:33:23 DEBUG }, 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG "0.005306153452956114" 09:33:23 DEBUG ] 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "instance": "10.142.15.199:9100" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "instance": "10.142.15.212:9100" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0.0025333333333333505" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG { 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG "data": { 09:33:23 DEBUG { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG "data": { 09:33:23 DEBUG { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "result": [ 09:33:23 DEBUG "pod": "ds-idrepo-0" 09:33:23 DEBUG { 09:33:23 DEBUG }, 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "value": [ 09:33:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:33:23 DEBUG 09:33:23 DEBUG }, 09:33:23 DEBUG 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "ds-cts-1" 09:33:23 DEBUG 09:33:23 DEBUG }, 09:33:23 DEBUG 09:33:23 DEBUG "value": [ 09:33:23 DEBUG "pod": "ds-cts-0" 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG }, 09:33:23 DEBUG "0" 09:33:23 DEBUG "value": [ 09:33:23 DEBUG ] 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG } 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "pod": "ds-idrepo-0" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "pod": "ds-idrepo-1" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "0" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262803" 09:33:23 INFO [http_cmd]: http status code OK 09:33:23 DEBUG --- status code --- 09:33:23 DEBUG http status code is 200 (expected 200) 09:33:23 DEBUG --- http response --- 09:33:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:33:23 DEBUG ----- output ----- 09:33:23 DEBUG { 09:33:23 DEBUG "status": "success", 09:33:23 DEBUG "data": { 09:33:23 DEBUG "resultType": "vector", 09:33:23 DEBUG "result": [ 09:33:23 DEBUG { 09:33:23 DEBUG "metric": { 09:33:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG [...] 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG 09:33:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:33:23 DEBUG }, 09:33:23 DEBUG "value": [ 09:33:23 DEBUG 1685262803, 09:33:23 DEBUG "3.2056789484314887" 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG ] 09:33:23 DEBUG } 09:33:23 DEBUG } 09:33:30 INFO 09:33:30 INFO [loop_until]: kubectl --namespace=xlou top pods 09:33:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:33:30 INFO [loop_until]: OK (rc = 0) 09:33:30 DEBUG --- stdout --- 09:33:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 10m 5814Mi am-7849cf7bdb-vcl2f 9m 5766Mi ds-cts-0 10m 455Mi ds-cts-1 9m 409Mi ds-cts-2 8m 426Mi ds-idrepo-0 1395m 9896Mi ds-idrepo-1 13m 13723Mi ds-idrepo-2 13m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 3105m 4364Mi idm-58dc667486-h86fb 252m 4061Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 841m 2695Mi 09:33:30 DEBUG --- stderr --- 09:33:30 DEBUG 09:33:32 INFO 09:33:32 INFO [loop_until]: kubectl --namespace=xlou top node 09:33:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:33:32 INFO [loop_until]: OK (rc = 0) 09:33:32 DEBUG --- stdout --- 09:33:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 75m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 73m 0% 6652Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 130m 0% 1538Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 82m 0% 6817Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 1286m 8% 5554Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 696m 4% 5210Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 86m 0% 1346Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1093Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 60m 0% 1100Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13160Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 78m 0% 10431Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14329Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 1179m 7% 3637Mi 6% 09:33:32 DEBUG --- stderr --- 09:33:32 DEBUG 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "ds-idrepo-2" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0.9091521314566637" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262863" 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "4635295744" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "ds-idrepo-0" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "ds-idrepo-1" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "ds-idrepo-0" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "ds-idrepo-1" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "ds-idrepo-0" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "ds-cts-1" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262863" 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "instance": "10.142.15.199:9100" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "instance": "10.142.15.212:9100" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "instance": "10.142.15.199:9100" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "instance": "10.142.15.212:9100" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "31279.233333333334" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0.2" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "ds-idrepo-2" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "8413.226479897636" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "ds-cts-2" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "ds-cts-1" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "instance": "10.142.15.199:9100" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "instance": "10.142.15.212:9100" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "28373.566666666666" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262863" 09:34:23 DEBUG 09:34:23 DEBUG "pod": "ds-idrepo-2" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0.013570280187229927" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "instance": "10.142.15.199:9100" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "instance": "10.142.15.212:9100" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0.0029333333333331287" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG { 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "data": { 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG "result": [ 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG { 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "pod": "ds-idrepo-0" 09:34:23 DEBUG "pod": "ds-idrepo-0" 09:34:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:34:23 DEBUG }, 09:34:23 DEBUG }, 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG "value": [ 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG [...] 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "pod": "ds-cts-1" 09:34:23 DEBUG "pod": "ds-idrepo-1" 09:34:23 DEBUG "pod": "ds-idrepo-2" 09:34:23 DEBUG }, 09:34:23 DEBUG }, 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG "value": [ 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0" 09:34:23 DEBUG "0" 09:34:23 DEBUG "0" 09:34:23 DEBUG ] 09:34:23 DEBUG ] 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG ] 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262863" 09:34:23 INFO [http_cmd]: http status code OK 09:34:23 DEBUG --- status code --- 09:34:23 DEBUG http status code is 200 (expected 200) 09:34:23 DEBUG --- http response --- 09:34:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:34:23 DEBUG ----- output ----- 09:34:23 DEBUG { 09:34:23 DEBUG "status": "success", 09:34:23 DEBUG "data": { 09:34:23 DEBUG "resultType": "vector", 09:34:23 DEBUG "result": [ 09:34:23 DEBUG { 09:34:23 DEBUG "metric": { 09:34:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG [...] 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG 09:34:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:34:23 DEBUG }, 09:34:23 DEBUG "value": [ 09:34:23 DEBUG 1685262863, 09:34:23 DEBUG "0.3752553638860443" 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG ] 09:34:23 DEBUG } 09:34:23 DEBUG } 09:34:30 INFO 09:34:30 INFO [loop_until]: kubectl --namespace=xlou top pods 09:34:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:34:30 INFO [loop_until]: OK (rc = 0) 09:34:30 DEBUG --- stdout --- 09:34:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5742Mi am-7849cf7bdb-cwb58 10m 5814Mi am-7849cf7bdb-vcl2f 9m 5766Mi ds-cts-0 8m 455Mi ds-cts-1 8m 409Mi ds-cts-2 9m 426Mi ds-idrepo-0 27m 9896Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 9m 4362Mi idm-58dc667486-h86fb 7m 4059Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 1m 341Mi 09:34:30 DEBUG --- stderr --- 09:34:30 DEBUG 09:34:32 INFO 09:34:32 INFO [loop_until]: kubectl --namespace=xlou top node 09:34:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:34:32 INFO [loop_until]: OK (rc = 0) 09:34:32 DEBUG --- stdout --- 09:34:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 93m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 74m 0% 6652Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 117m 0% 1544Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 84m 0% 6815Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 82m 0% 5556Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 80m 0% 5205Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 88m 0% 1349Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1093Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1103Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 56m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13162Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 86m 0% 10433Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14327Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 71m 0% 1297Mi 2% 09:34:32 DEBUG --- stderr --- 09:34:32 DEBUG 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG { 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG "data": { 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "result": [ 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "ds-idrepo-0" 09:35:23 DEBUG { 09:35:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:35:23 DEBUG }, 09:35:23 DEBUG "metric": { 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG }, 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "ds-idrepo-1" 09:35:23 DEBUG 09:35:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:35:23 DEBUG }, 09:35:23 DEBUG 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG }, 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0" 09:35:23 DEBUG "value": [ 09:35:23 DEBUG "0" 09:35:23 DEBUG ] 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG "4633542656" 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG ] 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "ds-idrepo-0" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "ds-idrepo-1" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "ds-idrepo-0" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "ds-cts-1" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262923" 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "instance": "10.142.15.199:9100" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "instance": "10.142.15.212:9100" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "instance": "10.142.15.199:9100" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "instance": "10.142.15.212:9100" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "31255.23333333333" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "ds-cts-2" 09:35:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:35:23 DEBUG }, 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "ds-cts-1" 09:35:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:35:23 DEBUG }, 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0" 09:35:23 DEBUG "0" 09:35:23 DEBUG ] 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:35:23 DEBUG }, 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0.2" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "626.5919179972402" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0.002724359754930349" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "instance": "10.142.15.199:9100" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "instance": "10.142.15.212:9100" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "28389.2" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:35:23 DEBUG { 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG { 09:35:23 DEBUG "data": { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "result": [ 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG { 09:35:23 DEBUG "result": [ 09:35:23 DEBUG "metric": { 09:35:23 DEBUG { 09:35:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:35:23 DEBUG "metric": { 09:35:23 DEBUG }, 09:35:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:35:23 DEBUG "value": [ 09:35:23 DEBUG }, 09:35:23 DEBUG 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:35:23 DEBUG 09:35:23 DEBUG }, 09:35:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:35:23 DEBUG "value": [ 09:35:23 DEBUG }, 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG "0" 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG ] 09:35:23 DEBUG "0" 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG { 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG } 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "instance": "10.142.15.199:9100" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "instance": "10.142.15.212:9100" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0.002433333333333347" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "ds-idrepo-0" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "ds-idrepo-1" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: http status code OK 09:35:23 DEBUG --- status code --- 09:35:23 DEBUG http status code is 200 (expected 200) 09:35:23 DEBUG --- http response --- 09:35:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:35:23 DEBUG ----- output ----- 09:35:23 DEBUG { 09:35:23 DEBUG "status": "success", 09:35:23 DEBUG "data": { 09:35:23 DEBUG "resultType": "vector", 09:35:23 DEBUG "result": [ 09:35:23 DEBUG { 09:35:23 DEBUG "metric": { 09:35:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG [...] 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG 09:35:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:35:23 DEBUG }, 09:35:23 DEBUG "value": [ 09:35:23 DEBUG 1685262923, 09:35:23 DEBUG "0" 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG ] 09:35:23 DEBUG } 09:35:23 DEBUG } 09:35:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262923" 09:35:24 INFO [http_cmd]: http status code OK 09:35:24 DEBUG --- status code --- 09:35:24 DEBUG http status code is 200 (expected 200) 09:35:24 DEBUG --- http response --- 09:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:35:24 DEBUG ----- output ----- 09:35:24 DEBUG { 09:35:24 DEBUG "status": "success", 09:35:24 DEBUG "data": { 09:35:24 DEBUG "resultType": "vector", 09:35:24 DEBUG "result": [ 09:35:24 DEBUG { 09:35:24 DEBUG "metric": { 09:35:24 DEBUG "pod": "ds-idrepo-0" 09:35:24 DEBUG }, 09:35:24 DEBUG "value": [ 09:35:24 DEBUG 09:35:24 DEBUG 09:35:24 DEBUG 09:35:24 DEBUG [...] 09:35:24 DEBUG 09:35:24 DEBUG 09:35:24 DEBUG 09:35:24 DEBUG "pod": "ds-cts-1" 09:35:24 DEBUG }, 09:35:24 DEBUG "value": [ 09:35:24 DEBUG 1685262923, 09:35:24 DEBUG "0" 09:35:24 DEBUG ] 09:35:24 DEBUG } 09:35:24 DEBUG ] 09:35:24 DEBUG } 09:35:24 DEBUG } 09:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262924" 09:35:24 INFO [http_cmd]: http status code OK 09:35:24 DEBUG --- status code --- 09:35:24 DEBUG http status code is 200 (expected 200) 09:35:24 DEBUG --- http response --- 09:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:35:24 DEBUG ----- output ----- 09:35:24 DEBUG { 09:35:24 DEBUG "status": "success", 09:35:24 DEBUG "data": { 09:35:24 DEBUG "resultType": "vector", 09:35:24 DEBUG "result": [ 09:35:24 DEBUG { 09:35:24 DEBUG "metric": { 09:35:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:35:24 DEBUG }, 09:35:24 DEBUG "value": [ 09:35:24 DEBUG 09:35:24 DEBUG 09:35:24 DEBUG 09:35:24 DEBUG [...] 09:35:24 DEBUG 09:35:24 DEBUG 09:35:24 DEBUG 09:35:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:35:24 DEBUG }, 09:35:24 DEBUG "value": [ 09:35:24 DEBUG 1685262924, 09:35:24 DEBUG "0.02796227360899705" 09:35:24 DEBUG ] 09:35:24 DEBUG } 09:35:24 DEBUG ] 09:35:24 DEBUG } 09:35:24 DEBUG } 09:35:30 INFO 09:35:30 INFO [loop_until]: kubectl --namespace=xlou top pods 09:35:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:35:30 INFO [loop_until]: OK (rc = 0) 09:35:30 DEBUG --- stdout --- 09:35:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 11m 5815Mi am-7849cf7bdb-vcl2f 8m 5766Mi ds-cts-0 9m 455Mi ds-cts-1 9m 409Mi ds-cts-2 9m 426Mi ds-idrepo-0 21m 9896Mi ds-idrepo-1 15m 13724Mi ds-idrepo-2 15m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 8m 4362Mi idm-58dc667486-h86fb 5m 4059Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3m 342Mi 09:35:30 DEBUG --- stderr --- 09:35:30 DEBUG 09:35:32 INFO 09:35:32 INFO [loop_until]: kubectl --namespace=xlou top node 09:35:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:35:32 INFO [loop_until]: OK (rc = 0) 09:35:32 DEBUG --- stdout --- 09:35:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 95m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 76m 0% 6651Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 128m 0% 1537Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 84m 0% 6815Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 83m 0% 5548Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 75m 0% 5206Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 84m 0% 1345Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1090Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1096Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1074Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 65m 0% 13160Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 80m 0% 10432Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14324Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 692m 4% 2078Mi 3% 09:35:32 DEBUG --- stderr --- 09:35:32 DEBUG 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG { 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "data": { 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "result": [ 09:36:23 DEBUG { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG [...] 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG "0" 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685262983" 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG { 09:36:23 DEBUG { 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "data": { 09:36:23 DEBUG "data": { 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "result": [ 09:36:23 DEBUG "result": [ 09:36:23 DEBUG { 09:36:23 DEBUG { 09:36:23 DEBUG "metric": { 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "pod": "ds-idrepo-0" 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:36:23 DEBUG }, 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG { 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG [...] 09:36:23 DEBUG [...] 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "data": { 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "result": [ 09:36:23 DEBUG { 09:36:23 DEBUG "pod": "ds-idrepo-1" 09:36:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:36:23 DEBUG { 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG }, 09:36:23 DEBUG }, 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "data": { 09:36:23 DEBUG "value": [ 09:36:23 DEBUG "value": [ 09:36:23 DEBUG "pod": "ds-idrepo-0" 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG }, 09:36:23 DEBUG "result": [ 09:36:23 DEBUG "0" 09:36:23 DEBUG "0.06666666666666667" 09:36:23 DEBUG "value": [ 09:36:23 DEBUG { 09:36:23 DEBUG ] 09:36:23 DEBUG ] 09:36:23 DEBUG 09:36:23 DEBUG "metric": { 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 DEBUG 09:36:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:36:23 DEBUG ] 09:36:23 DEBUG ] 09:36:23 DEBUG 09:36:23 DEBUG }, 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 DEBUG [...] 09:36:23 DEBUG "value": [ 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "pod": "ds-idrepo-1" 09:36:23 DEBUG [...] 09:36:23 DEBUG }, 09:36:23 DEBUG 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG 09:36:23 DEBUG "0" 09:36:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:36:23 DEBUG ] 09:36:23 DEBUG }, 09:36:23 DEBUG } 09:36:23 DEBUG "value": [ 09:36:23 DEBUG ] 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG } 09:36:23 DEBUG "4641361920" 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG { 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "data": { 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "result": [ 09:36:23 DEBUG { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "pod": "ds-idrepo-0" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG [...] 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "pod": "ds-cts-1" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG "0" 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262983" 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262983" 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG { 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "data": { 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "result": [ 09:36:23 DEBUG { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "instance": "10.142.15.199:9100" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG [...] 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "instance": "10.142.15.212:9100" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG "0.0006666666666660604" 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG { 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "data": { 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "result": [ 09:36:23 DEBUG { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "instance": "10.142.15.199:9100" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG [...] 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "instance": "10.142.15.212:9100" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG "31454.233333333334" 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG --- http response --- 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG { 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:36:23 DEBUG "data": { 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG { 09:36:23 DEBUG "result": [ 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG { 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG "data": { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "pod": "ds-cts-2" 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG "result": [ 09:36:23 DEBUG }, 09:36:23 DEBUG { 09:36:23 DEBUG { 09:36:23 DEBUG { 09:36:23 DEBUG { 09:36:23 DEBUG "value": [ 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "metric": { 09:36:23 DEBUG 09:36:23 DEBUG "data": { 09:36:23 DEBUG "data": { 09:36:23 DEBUG "data": { 09:36:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:36:23 DEBUG 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG }, 09:36:23 DEBUG 09:36:23 DEBUG "result": [ 09:36:23 DEBUG "result": [ 09:36:23 DEBUG "result": [ 09:36:23 DEBUG "value": [ 09:36:23 DEBUG [...] 09:36:23 DEBUG { 09:36:23 DEBUG { 09:36:23 DEBUG { 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:36:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:36:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG }, 09:36:23 DEBUG }, 09:36:23 DEBUG }, 09:36:23 DEBUG [...] 09:36:23 DEBUG "pod": "ds-cts-1" 09:36:23 DEBUG "value": [ 09:36:23 DEBUG "value": [ 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG }, 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:36:23 DEBUG "0" 09:36:23 DEBUG [...] 09:36:23 DEBUG [...] 09:36:23 DEBUG [...] 09:36:23 DEBUG }, 09:36:23 DEBUG ] 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "value": [ 09:36:23 DEBUG } 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG ] 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "7.03357780553492" 09:36:23 DEBUG } 09:36:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:36:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:36:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG }, 09:36:23 DEBUG }, 09:36:23 DEBUG }, 09:36:23 DEBUG } 09:36:23 DEBUG "value": [ 09:36:23 DEBUG "value": [ 09:36:23 DEBUG "value": [ 09:36:23 DEBUG ] 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG } 09:36:23 DEBUG "0.2" 09:36:23 DEBUG "459057.27501537616" 09:36:23 DEBUG "0.06666666666666667" 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG ] 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG ] 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262983" 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685262983" 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG { 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "data": { 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "result": [ 09:36:23 DEBUG { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "instance": "10.142.15.199:9100" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG [...] 09:36:23 DEBUG 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "instance": "10.142.15.212:9100" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG "28462.666666666664" 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG { 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "data": { 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "result": [ 09:36:23 DEBUG { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "instance": "10.142.15.199:9100" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG [...] 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "instance": "10.142.15.212:9100" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG "0.0027666666666668503" 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 INFO [http_cmd]: http status code OK 09:36:23 DEBUG --- status code --- 09:36:23 DEBUG http status code is 200 (expected 200) 09:36:23 DEBUG --- http response --- 09:36:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:36:23 DEBUG ----- output ----- 09:36:23 DEBUG { 09:36:23 DEBUG "status": "success", 09:36:23 DEBUG "data": { 09:36:23 DEBUG "resultType": "vector", 09:36:23 DEBUG "result": [ 09:36:23 DEBUG { 09:36:23 DEBUG "metric": { 09:36:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG [...] 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG 09:36:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:36:23 DEBUG }, 09:36:23 DEBUG "value": [ 09:36:23 DEBUG 1685262983, 09:36:23 DEBUG "0.03333333333333333" 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG ] 09:36:23 DEBUG } 09:36:23 DEBUG } 09:36:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262983" 09:36:24 INFO [http_cmd]: http status code OK 09:36:24 DEBUG --- status code --- 09:36:24 DEBUG http status code is 200 (expected 200) 09:36:24 DEBUG --- http response --- 09:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:36:24 DEBUG ----- output ----- 09:36:24 DEBUG { 09:36:24 DEBUG "status": "success", 09:36:24 DEBUG "data": { 09:36:24 DEBUG "resultType": "vector", 09:36:24 DEBUG "result": [ 09:36:24 DEBUG { 09:36:24 DEBUG "metric": { 09:36:24 DEBUG "pod": "ds-idrepo-0" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG [...] 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG "pod": "ds-idrepo-1" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 1685262983, 09:36:24 DEBUG "0" 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG } 09:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262984" 09:36:24 INFO [http_cmd]: http status code OK 09:36:24 DEBUG --- status code --- 09:36:24 DEBUG http status code is 200 (expected 200) 09:36:24 DEBUG --- http response --- 09:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:36:24 DEBUG ----- output ----- 09:36:24 DEBUG { 09:36:24 DEBUG "status": "success", 09:36:24 DEBUG "data": { 09:36:24 DEBUG "resultType": "vector", 09:36:24 DEBUG "result": [ 09:36:24 DEBUG { 09:36:24 DEBUG "metric": { 09:36:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG [...] 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 1685262984, 09:36:24 DEBUG "0.03333333333333333" 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG } 09:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262984" 09:36:24 INFO [http_cmd]: http status code OK 09:36:24 DEBUG --- status code --- 09:36:24 DEBUG http status code is 200 (expected 200) 09:36:24 DEBUG --- http response --- 09:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:36:24 DEBUG ----- output ----- 09:36:24 DEBUG { 09:36:24 DEBUG "status": "success", 09:36:24 DEBUG "data": { 09:36:24 DEBUG "resultType": "vector", 09:36:24 DEBUG "result": [ 09:36:24 DEBUG { 09:36:24 DEBUG "metric": { 09:36:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG [...] 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 1685262984, 09:36:24 DEBUG "0" 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG } 09:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685262984" 09:36:24 INFO [http_cmd]: http status code OK 09:36:24 DEBUG --- status code --- 09:36:24 DEBUG http status code is 200 (expected 200) 09:36:24 DEBUG --- http response --- 09:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:36:24 DEBUG ----- output ----- 09:36:24 DEBUG { 09:36:24 DEBUG "status": "success", 09:36:24 DEBUG "data": { 09:36:24 DEBUG "resultType": "vector", 09:36:24 DEBUG "result": [ 09:36:24 DEBUG { 09:36:24 DEBUG "metric": { 09:36:24 DEBUG "pod": "ds-idrepo-0" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG [...] 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG "pod": "ds-cts-1" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 1685262984, 09:36:24 DEBUG "0" 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG } 09:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685262984" 09:36:24 INFO [http_cmd]: http status code OK 09:36:24 DEBUG --- status code --- 09:36:24 DEBUG http status code is 200 (expected 200) 09:36:24 DEBUG --- http response --- 09:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:36:24 DEBUG ----- output ----- 09:36:24 DEBUG { 09:36:24 DEBUG "status": "success", 09:36:24 DEBUG "data": { 09:36:24 DEBUG "resultType": "vector", 09:36:24 DEBUG "result": [ 09:36:24 DEBUG { 09:36:24 DEBUG "metric": { 09:36:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG [...] 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG 09:36:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:36:24 DEBUG }, 09:36:24 DEBUG "value": [ 09:36:24 DEBUG 1685262984, 09:36:24 DEBUG "2.4453289742832065" 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG ] 09:36:24 DEBUG } 09:36:24 DEBUG } 09:36:30 INFO 09:36:30 INFO [loop_until]: kubectl --namespace=xlou top pods 09:36:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:36:30 INFO [loop_until]: OK (rc = 0) 09:36:30 DEBUG --- stdout --- 09:36:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 5Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 5m 5815Mi am-7849cf7bdb-vcl2f 9m 5766Mi ds-cts-0 8m 455Mi ds-cts-1 9m 409Mi ds-cts-2 8m 426Mi ds-idrepo-0 6900m 9896Mi ds-idrepo-1 15m 13724Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 9712m 4376Mi idm-58dc667486-h86fb 8896m 4072Mi lodemon-cfd7686d9-6dlt4 1m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3618m 2666Mi 09:36:30 DEBUG --- stderr --- 09:36:30 DEBUG 09:36:32 INFO 09:36:32 INFO [loop_until]: kubectl --namespace=xlou top node 09:36:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:36:32 INFO [loop_until]: OK (rc = 0) 09:36:32 DEBUG --- stdout --- 09:36:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 95m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 74m 0% 6651Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 131m 0% 1546Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 80m 0% 6811Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14370m 90% 5569Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9308m 58% 5219Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 82m 0% 1347Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1091Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1099Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1076Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13176Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6969m 43% 10433Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14326Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3614m 22% 3606Mi 6% 09:36:32 DEBUG --- stderr --- 09:36:32 DEBUG 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG "pod": "ds-cts-0" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG "0" 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263043" 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG "0" 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG { 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "data": { 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG "result": [ 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG { 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "pod": "ds-idrepo-0" 09:37:23 DEBUG "pod": "ds-idrepo-0" 09:37:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:37:23 DEBUG }, 09:37:23 DEBUG }, 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG "value": [ 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG [...] 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG "pod": "ds-idrepo-1" 09:37:23 DEBUG "pod": "ds-idrepo-1" 09:37:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:37:23 DEBUG }, 09:37:23 DEBUG }, 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG "value": [ 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG "0" 09:37:23 DEBUG "0" 09:37:23 DEBUG "4652793856" 09:37:23 DEBUG ] 09:37:23 DEBUG ] 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 DEBUG ] 09:37:23 DEBUG ] 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "pod": "ds-idrepo-0" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG "pod": "ds-cts-1" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG "0" 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263043" 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263043" 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "instance": "10.142.15.199:9100" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG "instance": "10.142.15.212:9100" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG "0.00033333333332545106" 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "instance": "10.142.15.199:9100" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG "instance": "10.142.15.212:9100" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG "31286.133333333335" 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG "pod": "ds-cts-0" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG "0.004020478010355821" 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG "0" 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263043" 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "pod": "ds-cts-2" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG "pod": "ds-cts-1" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 1685263043, 09:37:23 DEBUG "0" 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG ] 09:37:23 DEBUG } 09:37:23 DEBUG } 09:37:23 INFO [http_cmd]: http status code OK 09:37:23 DEBUG --- status code --- 09:37:23 DEBUG http status code is 200 (expected 200) 09:37:23 DEBUG --- http response --- 09:37:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:37:23 DEBUG ----- output ----- 09:37:23 DEBUG { 09:37:23 DEBUG "status": "success", 09:37:23 DEBUG "data": { 09:37:23 DEBUG "resultType": "vector", 09:37:23 DEBUG "result": [ 09:37:23 DEBUG { 09:37:23 DEBUG "metric": { 09:37:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:37:23 DEBUG }, 09:37:23 DEBUG "value": [ 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG 09:37:23 DEBUG [...] 09:37:23 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 1685263043, 09:37:24 DEBUG "0.2" 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG } 09:37:24 INFO [http_cmd]: http status code OK 09:37:24 DEBUG --- status code --- 09:37:24 DEBUG http status code is 200 (expected 200) 09:37:24 DEBUG --- http response --- 09:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:37:24 DEBUG ----- output ----- 09:37:24 DEBUG { 09:37:24 DEBUG "status": "success", 09:37:24 DEBUG "data": { 09:37:24 DEBUG "resultType": "vector", 09:37:24 DEBUG "result": [ 09:37:24 DEBUG { 09:37:24 DEBUG "metric": { 09:37:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG [...] 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "pod": "ds-cts-0" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 1685263043, 09:37:24 DEBUG "2480.7887982270577" 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG } 09:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263044" 09:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263044" 09:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263044" 09:37:24 INFO [http_cmd]: http status code OK 09:37:24 DEBUG --- status code --- 09:37:24 DEBUG http status code is 200 (expected 200) 09:37:24 INFO [http_cmd]: http status code OK 09:37:24 DEBUG --- http response --- 09:37:24 DEBUG --- status code --- 09:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:37:24 DEBUG http status code is 200 (expected 200) 09:37:24 DEBUG ----- output ----- 09:37:24 DEBUG --- http response --- 09:37:24 DEBUG { 09:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:37:24 DEBUG "status": "success", 09:37:24 DEBUG ----- output ----- 09:37:24 DEBUG "data": { 09:37:24 INFO [http_cmd]: http status code OK 09:37:24 DEBUG { 09:37:24 DEBUG "resultType": "vector", 09:37:24 DEBUG --- status code --- 09:37:24 DEBUG "status": "success", 09:37:24 DEBUG "result": [ 09:37:24 DEBUG http status code is 200 (expected 200) 09:37:24 DEBUG "data": { 09:37:24 DEBUG { 09:37:24 DEBUG --- http response --- 09:37:24 DEBUG "resultType": "vector", 09:37:24 DEBUG "metric": { 09:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:37:24 DEBUG "result": [ 09:37:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:37:24 DEBUG ----- output ----- 09:37:24 DEBUG { 09:37:24 DEBUG }, 09:37:24 DEBUG { 09:37:24 DEBUG "metric": { 09:37:24 DEBUG "value": [ 09:37:24 DEBUG "status": "success", 09:37:24 DEBUG "instance": "10.142.15.199:9100" 09:37:24 DEBUG 09:37:24 DEBUG "data": { 09:37:24 DEBUG }, 09:37:24 DEBUG 09:37:24 DEBUG "resultType": "vector", 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 09:37:24 DEBUG "result": [ 09:37:24 DEBUG 09:37:24 DEBUG [...] 09:37:24 DEBUG { 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "metric": { 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "instance": "10.142.15.199:9100" 09:37:24 DEBUG [...] 09:37:24 DEBUG 09:37:24 DEBUG }, 09:37:24 DEBUG 09:37:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 09:37:24 DEBUG }, 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 09:37:24 DEBUG "instance": "10.142.15.212:9100" 09:37:24 DEBUG 1685263044, 09:37:24 DEBUG 09:37:24 DEBUG }, 09:37:24 DEBUG "0" 09:37:24 DEBUG [...] 09:37:24 DEBUG "value": [ 09:37:24 DEBUG ] 09:37:24 DEBUG 09:37:24 DEBUG 1685263044, 09:37:24 DEBUG } 09:37:24 DEBUG 09:37:24 DEBUG "28215.366666666665" 09:37:24 DEBUG ] 09:37:24 DEBUG 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG "instance": "10.142.15.212:9100" 09:37:24 DEBUG } 09:37:24 DEBUG } 09:37:24 DEBUG }, 09:37:24 DEBUG ] 09:37:24 DEBUG "value": [ 09:37:24 DEBUG } 09:37:24 DEBUG 1685263044, 09:37:24 DEBUG } 09:37:24 DEBUG "0.002800000000000343" 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG } 09:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263044" 09:37:24 INFO [http_cmd]: http status code OK 09:37:24 DEBUG --- status code --- 09:37:24 DEBUG http status code is 200 (expected 200) 09:37:24 DEBUG --- http response --- 09:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:37:24 DEBUG ----- output ----- 09:37:24 DEBUG { 09:37:24 DEBUG "status": "success", 09:37:24 DEBUG "data": { 09:37:24 DEBUG "resultType": "vector", 09:37:24 DEBUG "result": [ 09:37:24 DEBUG { 09:37:24 DEBUG "metric": { 09:37:24 DEBUG "pod": "ds-idrepo-0" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG [...] 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "pod": "ds-idrepo-1" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 1685263044, 09:37:24 DEBUG "0" 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG } 09:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263044" 09:37:24 INFO [http_cmd]: http status code OK 09:37:24 DEBUG --- status code --- 09:37:24 DEBUG http status code is 200 (expected 200) 09:37:24 DEBUG --- http response --- 09:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:37:24 DEBUG ----- output ----- 09:37:24 DEBUG { 09:37:24 DEBUG "status": "success", 09:37:24 DEBUG "data": { 09:37:24 DEBUG "resultType": "vector", 09:37:24 DEBUG "result": [ 09:37:24 DEBUG { 09:37:24 DEBUG "metric": { 09:37:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG [...] 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 1685263044, 09:37:24 DEBUG "0" 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG } 09:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263044" 09:37:24 INFO [http_cmd]: http status code OK 09:37:24 DEBUG --- status code --- 09:37:24 DEBUG http status code is 200 (expected 200) 09:37:24 DEBUG --- http response --- 09:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:37:24 DEBUG ----- output ----- 09:37:24 DEBUG { 09:37:24 DEBUG "status": "success", 09:37:24 DEBUG "data": { 09:37:24 DEBUG "resultType": "vector", 09:37:24 DEBUG "result": [ 09:37:24 DEBUG { 09:37:24 DEBUG "metric": { 09:37:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG [...] 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 1685263044, 09:37:24 DEBUG "0" 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG } 09:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263044" 09:37:24 INFO [http_cmd]: http status code OK 09:37:24 DEBUG --- status code --- 09:37:24 DEBUG http status code is 200 (expected 200) 09:37:24 DEBUG --- http response --- 09:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:37:24 DEBUG ----- output ----- 09:37:24 DEBUG { 09:37:24 DEBUG "status": "success", 09:37:24 DEBUG "data": { 09:37:24 DEBUG "resultType": "vector", 09:37:24 DEBUG "result": [ 09:37:24 DEBUG { 09:37:24 DEBUG "metric": { 09:37:24 DEBUG "pod": "ds-idrepo-0" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG [...] 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "pod": "ds-cts-1" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 1685263044, 09:37:24 DEBUG "0" 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG } 09:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263044" 09:37:24 INFO [http_cmd]: http status code OK 09:37:24 DEBUG --- status code --- 09:37:24 DEBUG http status code is 200 (expected 200) 09:37:24 DEBUG --- http response --- 09:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:37:24 DEBUG ----- output ----- 09:37:24 DEBUG { 09:37:24 DEBUG "status": "success", 09:37:24 DEBUG "data": { 09:37:24 DEBUG "resultType": "vector", 09:37:24 DEBUG "result": [ 09:37:24 DEBUG { 09:37:24 DEBUG "metric": { 09:37:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG [...] 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG 09:37:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:37:24 DEBUG }, 09:37:24 DEBUG "value": [ 09:37:24 DEBUG 1685263044, 09:37:24 DEBUG "3.3742715668531083" 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG ] 09:37:24 DEBUG } 09:37:24 DEBUG } 09:37:30 INFO 09:37:30 INFO [loop_until]: kubectl --namespace=xlou top pods 09:37:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:37:30 INFO [loop_until]: OK (rc = 0) 09:37:30 DEBUG --- stdout --- 09:37:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 5m 5815Mi am-7849cf7bdb-vcl2f 9m 5766Mi ds-cts-0 9m 455Mi ds-cts-1 9m 409Mi ds-cts-2 8m 426Mi ds-idrepo-0 7175m 9897Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10381m 4446Mi idm-58dc667486-h86fb 9372m 4109Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3394m 2672Mi 09:37:30 DEBUG --- stderr --- 09:37:30 DEBUG 09:37:32 INFO 09:37:32 INFO [loop_until]: kubectl --namespace=xlou top node 09:37:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:37:32 INFO [loop_until]: OK (rc = 0) 09:37:32 DEBUG --- stdout --- 09:37:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 94m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 77m 0% 6662Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 118m 0% 1547Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 81m 0% 6813Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14702m 92% 5633Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9526m 59% 5227Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 81m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1090Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1099Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1078Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 65m 0% 13164Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6894m 43% 10431Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 64m 0% 14329Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3582m 22% 3612Mi 6% 09:37:32 DEBUG --- stderr --- 09:37:32 DEBUG 09:38:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263103" 09:38:23 INFO [http_cmd]: http status code OK 09:38:23 DEBUG --- status code --- 09:38:23 DEBUG http status code is 200 (expected 200) 09:38:23 DEBUG --- http response --- 09:38:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:38:23 DEBUG ----- output ----- 09:38:23 DEBUG { 09:38:23 DEBUG "status": "success", 09:38:23 DEBUG "data": { 09:38:23 DEBUG "resultType": "vector", 09:38:23 DEBUG "result": [ 09:38:23 DEBUG { 09:38:23 DEBUG "metric": { 09:38:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG [...] 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 1685263103, 09:38:23 DEBUG "0" 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG } 09:38:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263103" 09:38:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263103" 09:38:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263103" 09:38:23 INFO [http_cmd]: http status code OK 09:38:23 DEBUG --- status code --- 09:38:23 DEBUG http status code is 200 (expected 200) 09:38:23 DEBUG --- http response --- 09:38:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:38:23 DEBUG ----- output ----- 09:38:23 DEBUG { 09:38:23 DEBUG "status": "success", 09:38:23 DEBUG "data": { 09:38:23 DEBUG "resultType": "vector", 09:38:23 DEBUG "result": [ 09:38:23 DEBUG { 09:38:23 DEBUG "metric": { 09:38:23 DEBUG "pod": "ds-idrepo-0" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG [...] 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG "pod": "ds-idrepo-1" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 1685263103, 09:38:23 DEBUG "0" 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG } 09:38:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263103" 09:38:23 INFO [http_cmd]: http status code OK 09:38:23 DEBUG --- status code --- 09:38:23 DEBUG http status code is 200 (expected 200) 09:38:23 DEBUG --- http response --- 09:38:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:38:23 DEBUG ----- output ----- 09:38:23 DEBUG { 09:38:23 DEBUG "status": "success", 09:38:23 DEBUG "data": { 09:38:23 DEBUG "resultType": "vector", 09:38:23 DEBUG "result": [ 09:38:23 DEBUG { 09:38:23 DEBUG "metric": { 09:38:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG [...] 09:38:23 DEBUG 09:38:23 INFO [http_cmd]: http status code OK 09:38:23 DEBUG 09:38:23 DEBUG --- status code --- 09:38:23 DEBUG 09:38:23 DEBUG http status code is 200 (expected 200) 09:38:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:38:23 DEBUG --- http response --- 09:38:23 DEBUG }, 09:38:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:38:23 DEBUG "value": [ 09:38:23 DEBUG ----- output ----- 09:38:23 DEBUG 1685263103, 09:38:23 DEBUG { 09:38:23 DEBUG "0" 09:38:23 DEBUG "status": "success", 09:38:23 DEBUG ] 09:38:23 DEBUG "data": { 09:38:23 DEBUG } 09:38:23 DEBUG "resultType": "vector", 09:38:23 DEBUG ] 09:38:23 DEBUG "result": [ 09:38:23 DEBUG } 09:38:23 DEBUG { 09:38:23 DEBUG } 09:38:23 DEBUG "metric": { 09:38:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG [...] 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 1685263103, 09:38:23 DEBUG "4664733696" 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG } 09:38:23 INFO [http_cmd]: http status code OK 09:38:23 DEBUG --- status code --- 09:38:23 DEBUG http status code is 200 (expected 200) 09:38:23 DEBUG --- http response --- 09:38:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:38:23 DEBUG ----- output ----- 09:38:23 DEBUG { 09:38:23 DEBUG "status": "success", 09:38:23 DEBUG "data": { 09:38:23 DEBUG "resultType": "vector", 09:38:23 DEBUG "result": [ 09:38:23 DEBUG { 09:38:23 DEBUG "metric": { 09:38:23 DEBUG "pod": "ds-idrepo-0" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG [...] 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG "pod": "ds-idrepo-1" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 1685263103, 09:38:23 DEBUG "0" 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG } 09:38:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263103" 09:38:23 INFO [http_cmd]: http status code OK 09:38:23 DEBUG --- status code --- 09:38:23 DEBUG http status code is 200 (expected 200) 09:38:23 DEBUG --- http response --- 09:38:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:38:23 DEBUG ----- output ----- 09:38:23 DEBUG { 09:38:23 DEBUG "status": "success", 09:38:23 DEBUG "data": { 09:38:23 DEBUG "resultType": "vector", 09:38:23 DEBUG "result": [ 09:38:23 DEBUG { 09:38:23 DEBUG "metric": { 09:38:23 DEBUG "pod": "ds-idrepo-0" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG [...] 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG 09:38:23 DEBUG "pod": "ds-cts-1" 09:38:23 DEBUG }, 09:38:23 DEBUG "value": [ 09:38:23 DEBUG 1685263103, 09:38:23 DEBUG "0" 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG ] 09:38:23 DEBUG } 09:38:23 DEBUG } 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263104" 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263104" 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG --- status code --- 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:38:24 DEBUG { 09:38:24 DEBUG { 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG { 09:38:24 DEBUG "data": { 09:38:24 DEBUG "data": { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "result": [ 09:38:24 DEBUG "result": [ 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG { 09:38:24 DEBUG { 09:38:24 DEBUG "result": [ 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG { 09:38:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:38:24 DEBUG "instance": "10.142.15.199:9100" 09:38:24 DEBUG "metric": { 09:38:24 DEBUG }, 09:38:24 DEBUG }, 09:38:24 DEBUG "instance": "10.142.15.199:9100" 09:38:24 DEBUG "value": [ 09:38:24 DEBUG "value": [ 09:38:24 DEBUG }, 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG "pod": "ds-cts-0" 09:38:24 DEBUG "instance": "10.142.15.212:9100" 09:38:24 DEBUG 09:38:24 DEBUG }, 09:38:24 DEBUG }, 09:38:24 DEBUG "instance": "10.142.15.212:9100" 09:38:24 DEBUG "value": [ 09:38:24 DEBUG "value": [ 09:38:24 DEBUG }, 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG "0" 09:38:24 DEBUG "31352.966666666664" 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG ] 09:38:24 DEBUG ] 09:38:24 DEBUG "0.0010000000000066696" 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG ] 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG --- status code --- 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG { 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "result": [ 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "result": [ 09:38:24 DEBUG { 09:38:24 DEBUG "result": [ 09:38:24 DEBUG { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "pod": "ds-cts-2" 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:38:24 DEBUG }, 09:38:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG "pod": "ds-cts-1" 09:38:24 DEBUG 09:38:24 DEBUG "pod": "ds-cts-0" 09:38:24 DEBUG }, 09:38:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "0" 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "0" 09:38:24 DEBUG ] 09:38:24 DEBUG "0" 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "result": [ 09:38:24 DEBUG { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "0.2" 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "result": [ 09:38:24 DEBUG { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "pod": "ds-idrepo-0" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG "pod": "ds-idrepo-1" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "0" 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263104" 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263104" 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG { 09:38:24 DEBUG { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "data": { 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "result": [ 09:38:24 DEBUG "result": [ 09:38:24 DEBUG { 09:38:24 DEBUG { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:38:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:38:24 DEBUG }, 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG }, 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG }, 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG "value": [ 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG "value": [ 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:38:24 DEBUG "0" 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG "0" 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG ] 09:38:24 DEBUG { 09:38:24 DEBUG ] 09:38:24 DEBUG { 09:38:24 DEBUG } 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG } 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG ] 09:38:24 DEBUG "data": { 09:38:24 DEBUG ] 09:38:24 DEBUG "data": { 09:38:24 DEBUG } 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG } 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG } 09:38:24 DEBUG "result": [ 09:38:24 DEBUG } 09:38:24 DEBUG "result": [ 09:38:24 DEBUG { 09:38:24 DEBUG { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "instance": "10.142.15.199:9100" 09:38:24 DEBUG "instance": "10.142.15.199:9100" 09:38:24 DEBUG }, 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG "instance": "10.142.15.212:9100" 09:38:24 DEBUG "instance": "10.142.15.212:9100" 09:38:24 DEBUG }, 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "30288.76666666667" 09:38:24 DEBUG "0.0023333333333335794" 09:38:24 DEBUG ] 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "result": [ 09:38:24 DEBUG { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "pod": "ds-idrepo-0" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG "pod": "ds-cts-1" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "0" 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263104" 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "result": [ 09:38:24 DEBUG { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "0" 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263104" 09:38:24 INFO [http_cmd]: http status code OK 09:38:24 DEBUG --- status code --- 09:38:24 DEBUG http status code is 200 (expected 200) 09:38:24 DEBUG --- http response --- 09:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:38:24 DEBUG ----- output ----- 09:38:24 DEBUG { 09:38:24 DEBUG "status": "success", 09:38:24 DEBUG "data": { 09:38:24 DEBUG "resultType": "vector", 09:38:24 DEBUG "result": [ 09:38:24 DEBUG { 09:38:24 DEBUG "metric": { 09:38:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG [...] 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG 09:38:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:38:24 DEBUG }, 09:38:24 DEBUG "value": [ 09:38:24 DEBUG 1685263104, 09:38:24 DEBUG "3.4332359723128976" 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG ] 09:38:24 DEBUG } 09:38:24 DEBUG } 09:38:30 INFO 09:38:30 INFO [loop_until]: kubectl --namespace=xlou top pods 09:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:38:30 INFO [loop_until]: OK (rc = 0) 09:38:30 DEBUG --- stdout --- 09:38:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 6m 5815Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 8m 455Mi ds-cts-1 8m 409Mi ds-cts-2 8m 426Mi ds-idrepo-0 6978m 9896Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 15m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10437m 4455Mi idm-58dc667486-h86fb 8857m 4094Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3349m 2672Mi 09:38:30 DEBUG --- stderr --- 09:38:30 DEBUG 09:38:32 INFO 09:38:32 INFO [loop_until]: kubectl --namespace=xlou top node 09:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:38:32 INFO [loop_until]: OK (rc = 0) 09:38:32 DEBUG --- stdout --- 09:38:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 95m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 69m 0% 6649Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 128m 0% 1545Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 81m 0% 6828Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14736m 92% 5637Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9184m 57% 5236Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 82m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1094Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1098Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13165Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7058m 44% 10431Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 64m 0% 14326Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3499m 22% 3615Mi 6% 09:38:32 DEBUG --- stderr --- 09:38:32 DEBUG 09:39:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263163" 09:39:23 INFO [http_cmd]: http status code OK 09:39:23 DEBUG --- status code --- 09:39:23 DEBUG http status code is 200 (expected 200) 09:39:23 DEBUG --- http response --- 09:39:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:39:23 DEBUG ----- output ----- 09:39:23 DEBUG { 09:39:23 DEBUG "status": "success", 09:39:23 DEBUG "data": { 09:39:23 DEBUG "resultType": "vector", 09:39:23 DEBUG "result": [ 09:39:23 DEBUG { 09:39:23 DEBUG "metric": { 09:39:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:39:23 DEBUG }, 09:39:23 DEBUG "value": [ 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG [...] 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:39:23 DEBUG }, 09:39:23 DEBUG "value": [ 09:39:23 DEBUG 1685263163, 09:39:23 DEBUG "0" 09:39:23 DEBUG ] 09:39:23 DEBUG } 09:39:23 DEBUG ] 09:39:23 DEBUG } 09:39:23 DEBUG } 09:39:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263163" 09:39:23 INFO [http_cmd]: http status code OK 09:39:23 DEBUG --- status code --- 09:39:23 DEBUG http status code is 200 (expected 200) 09:39:23 DEBUG --- http response --- 09:39:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:39:23 DEBUG ----- output ----- 09:39:23 DEBUG { 09:39:23 DEBUG "status": "success", 09:39:23 DEBUG "data": { 09:39:23 DEBUG "resultType": "vector", 09:39:23 DEBUG "result": [ 09:39:23 DEBUG { 09:39:23 DEBUG "metric": { 09:39:23 DEBUG "pod": "ds-idrepo-0" 09:39:23 DEBUG }, 09:39:23 DEBUG "value": [ 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG [...] 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG "pod": "ds-idrepo-1" 09:39:23 DEBUG }, 09:39:23 DEBUG "value": [ 09:39:23 DEBUG 1685263163, 09:39:23 DEBUG "0" 09:39:23 DEBUG ] 09:39:23 DEBUG } 09:39:23 DEBUG ] 09:39:23 DEBUG } 09:39:23 DEBUG } 09:39:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263163" 09:39:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263163" 09:39:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263163" 09:39:23 INFO [http_cmd]: http status code OK 09:39:23 DEBUG --- status code --- 09:39:23 DEBUG http status code is 200 (expected 200) 09:39:23 DEBUG --- http response --- 09:39:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:39:23 DEBUG ----- output ----- 09:39:23 DEBUG { 09:39:23 DEBUG "status": "success", 09:39:23 DEBUG "data": { 09:39:23 DEBUG "resultType": "vector", 09:39:23 DEBUG "result": [ 09:39:23 DEBUG { 09:39:23 DEBUG "metric": { 09:39:23 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:39:23 DEBUG }, 09:39:23 DEBUG "value": [ 09:39:23 DEBUG 09:39:23 INFO [http_cmd]: http status code OK 09:39:23 DEBUG 09:39:23 DEBUG --- status code --- 09:39:23 DEBUG 09:39:23 DEBUG http status code is 200 (expected 200) 09:39:23 DEBUG [...] 09:39:23 DEBUG --- http response --- 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:39:23 INFO [http_cmd]: http status code OK 09:39:23 DEBUG 09:39:23 DEBUG ----- output ----- 09:39:23 DEBUG --- status code --- 09:39:23 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:39:23 DEBUG { 09:39:23 DEBUG http status code is 200 (expected 200) 09:39:23 DEBUG }, 09:39:23 DEBUG "status": "success", 09:39:23 DEBUG --- http response --- 09:39:23 DEBUG "value": [ 09:39:23 DEBUG "data": { 09:39:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:39:23 DEBUG 1685263163, 09:39:23 DEBUG "resultType": "vector", 09:39:23 DEBUG ----- output ----- 09:39:23 DEBUG "0" 09:39:23 DEBUG "result": [ 09:39:23 DEBUG { 09:39:23 DEBUG ] 09:39:23 DEBUG { 09:39:23 DEBUG "status": "success", 09:39:23 DEBUG } 09:39:23 DEBUG "metric": { 09:39:23 DEBUG "data": { 09:39:23 DEBUG ] 09:39:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:39:23 DEBUG "resultType": "vector", 09:39:23 DEBUG } 09:39:23 DEBUG }, 09:39:23 DEBUG "result": [ 09:39:23 DEBUG } 09:39:23 DEBUG "value": [ 09:39:23 DEBUG { 09:39:23 DEBUG 09:39:23 DEBUG "metric": { 09:39:23 DEBUG 09:39:23 DEBUG "pod": "ds-idrepo-0" 09:39:23 DEBUG 09:39:23 DEBUG }, 09:39:23 DEBUG [...] 09:39:23 DEBUG "value": [ 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG 09:39:23 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:39:23 DEBUG [...] 09:39:23 DEBUG }, 09:39:23 DEBUG 09:39:23 DEBUG "value": [ 09:39:23 DEBUG 09:39:23 DEBUG 1685263163, 09:39:23 DEBUG 09:39:23 DEBUG "4672876544" 09:39:23 DEBUG "pod": "ds-idrepo-1" 09:39:23 DEBUG ] 09:39:23 DEBUG }, 09:39:23 DEBUG } 09:39:23 DEBUG "value": [ 09:39:23 DEBUG ] 09:39:23 DEBUG 1685263163, 09:39:23 DEBUG } 09:39:23 DEBUG "0" 09:39:23 DEBUG } 09:39:23 DEBUG ] 09:39:23 DEBUG } 09:39:23 DEBUG ] 09:39:23 DEBUG } 09:39:23 DEBUG } 09:39:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263163" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "pod": "ds-idrepo-0" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "pod": "ds-cts-1" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263163, 09:39:24 DEBUG "0" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "instance": "10.142.15.199:9100" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "instance": "10.142.15.212:9100" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "31140.466666666667" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG "data": { 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG "result": [ 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG { 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:39:24 DEBUG "metric": { 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:39:24 DEBUG { 09:39:24 DEBUG }, 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "value": [ 09:39:24 DEBUG "data": { 09:39:24 DEBUG 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG 09:39:24 DEBUG "result": [ 09:39:24 DEBUG 09:39:24 DEBUG { 09:39:24 DEBUG [...] 09:39:24 DEBUG "metric": { 09:39:24 DEBUG 09:39:24 DEBUG "instance": "10.142.15.199:9100" 09:39:24 DEBUG 09:39:24 DEBUG }, 09:39:24 DEBUG 09:39:24 DEBUG "value": [ 09:39:24 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:39:24 DEBUG 09:39:24 DEBUG }, 09:39:24 DEBUG 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG [...] 09:39:24 DEBUG "6.903287810199796" 09:39:24 DEBUG 09:39:24 DEBUG ] 09:39:24 DEBUG 09:39:24 DEBUG } 09:39:24 DEBUG 09:39:24 DEBUG ] 09:39:24 DEBUG "instance": "10.142.15.212:9100" 09:39:24 DEBUG } 09:39:24 DEBUG }, 09:39:24 DEBUG } 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "0.0009999999999915114" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "0" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:39:24 DEBUG { 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG { 09:39:24 DEBUG "data": { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "result": [ 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG { 09:39:24 DEBUG "result": [ 09:39:24 DEBUG "metric": { 09:39:24 DEBUG { 09:39:24 DEBUG "pod": "ds-cts-2" 09:39:24 DEBUG "metric": { 09:39:24 DEBUG }, 09:39:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:39:24 DEBUG "value": [ 09:39:24 DEBUG }, 09:39:24 DEBUG 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "pod": "ds-cts-1" 09:39:24 DEBUG 09:39:24 DEBUG }, 09:39:24 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:39:24 DEBUG "value": [ 09:39:24 DEBUG }, 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG "0" 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG ] 09:39:24 DEBUG "446623.4142298224" 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "0.2" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "pod": "ds-idrepo-0" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "pod": "ds-idrepo-1" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "0" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "instance": "10.142.15.199:9100" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "instance": "10.142.15.212:9100" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "0.0026999999999996285" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "0" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:39:24 DEBUG "instance": "10.142.15.199:9100" 09:39:24 DEBUG }, 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:39:24 DEBUG "instance": "10.142.15.212:9100" 09:39:24 DEBUG }, 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "0" 09:39:24 DEBUG "28276.7" 09:39:24 DEBUG ] 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "pod": "ds-idrepo-0" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "pod": "ds-cts-1" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "0" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "0" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263164" 09:39:24 INFO [http_cmd]: http status code OK 09:39:24 DEBUG --- status code --- 09:39:24 DEBUG http status code is 200 (expected 200) 09:39:24 DEBUG --- http response --- 09:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:39:24 DEBUG ----- output ----- 09:39:24 DEBUG { 09:39:24 DEBUG "status": "success", 09:39:24 DEBUG "data": { 09:39:24 DEBUG "resultType": "vector", 09:39:24 DEBUG "result": [ 09:39:24 DEBUG { 09:39:24 DEBUG "metric": { 09:39:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG [...] 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG 09:39:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:39:24 DEBUG }, 09:39:24 DEBUG "value": [ 09:39:24 DEBUG 1685263164, 09:39:24 DEBUG "3.391990980089895" 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG ] 09:39:24 DEBUG } 09:39:24 DEBUG } 09:39:30 INFO 09:39:30 INFO [loop_until]: kubectl --namespace=xlou top pods 09:39:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:39:31 INFO [loop_until]: OK (rc = 0) 09:39:31 DEBUG --- stdout --- 09:39:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5742Mi am-7849cf7bdb-cwb58 5m 5815Mi am-7849cf7bdb-vcl2f 6m 5768Mi ds-cts-0 9m 455Mi ds-cts-1 9m 409Mi ds-cts-2 9m 426Mi ds-idrepo-0 6849m 9897Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10180m 4461Mi idm-58dc667486-h86fb 9246m 4101Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3477m 2674Mi 09:39:31 DEBUG --- stderr --- 09:39:31 DEBUG 09:39:32 INFO 09:39:32 INFO [loop_until]: kubectl --namespace=xlou top node 09:39:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:39:32 INFO [loop_until]: OK (rc = 0) 09:39:32 DEBUG --- stdout --- 09:39:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 98m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6655Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 130m 0% 1540Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6818Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14786m 93% 5644Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9378m 59% 5243Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1346Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1091Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1099Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 61m 0% 1072Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13166Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7043m 44% 10432Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 63m 0% 14329Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3578m 22% 3617Mi 6% 09:39:32 DEBUG --- stderr --- 09:39:32 DEBUG 09:40:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263223" 09:40:23 INFO [http_cmd]: http status code OK 09:40:23 DEBUG --- status code --- 09:40:23 DEBUG http status code is 200 (expected 200) 09:40:23 DEBUG --- http response --- 09:40:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:23 DEBUG ----- output ----- 09:40:23 DEBUG { 09:40:23 DEBUG "status": "success", 09:40:23 DEBUG "data": { 09:40:23 DEBUG "resultType": "vector", 09:40:23 DEBUG "result": [ 09:40:23 DEBUG { 09:40:23 DEBUG "metric": { 09:40:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:40:23 DEBUG }, 09:40:23 DEBUG "value": [ 09:40:23 DEBUG 09:40:23 DEBUG 09:40:23 DEBUG 09:40:23 DEBUG [...] 09:40:23 DEBUG 09:40:23 DEBUG 09:40:23 DEBUG 09:40:23 DEBUG "pod": "ds-idrepo-2" 09:40:23 DEBUG }, 09:40:23 DEBUG "value": [ 09:40:23 DEBUG 1685263223, 09:40:23 DEBUG "0.9280742459396752" 09:40:23 DEBUG ] 09:40:23 DEBUG } 09:40:23 DEBUG ] 09:40:23 DEBUG } 09:40:23 DEBUG } 09:40:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263223" 09:40:23 INFO [http_cmd]: http status code OK 09:40:23 DEBUG --- status code --- 09:40:23 DEBUG http status code is 200 (expected 200) 09:40:23 DEBUG --- http response --- 09:40:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:40:23 DEBUG ----- output ----- 09:40:23 DEBUG { 09:40:23 DEBUG "status": "success", 09:40:23 DEBUG "data": { 09:40:23 DEBUG "resultType": "vector", 09:40:23 DEBUG "result": [ 09:40:23 DEBUG { 09:40:23 DEBUG "metric": { 09:40:23 DEBUG "pod": "ds-idrepo-0" 09:40:23 DEBUG }, 09:40:23 DEBUG "value": [ 09:40:23 DEBUG 09:40:23 DEBUG 09:40:23 DEBUG 09:40:23 DEBUG [...] 09:40:23 DEBUG 09:40:23 DEBUG 09:40:23 DEBUG 09:40:23 DEBUG "pod": "ds-idrepo-1" 09:40:23 DEBUG }, 09:40:23 DEBUG "value": [ 09:40:23 DEBUG 1685263223, 09:40:23 DEBUG "0" 09:40:23 DEBUG ] 09:40:23 DEBUG } 09:40:23 DEBUG ] 09:40:23 DEBUG } 09:40:23 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263224" 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG "data": { 09:40:24 DEBUG { 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG "result": [ 09:40:24 DEBUG "data": { 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:24 DEBUG { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "pod": "ds-idrepo-0" 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG }, 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "data": { 09:40:24 DEBUG "value": [ 09:40:24 DEBUG "pod": "ds-idrepo-0" 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG 09:40:24 DEBUG }, 09:40:24 DEBUG "result": [ 09:40:24 DEBUG 09:40:24 DEBUG "value": [ 09:40:24 DEBUG { 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "metric": { 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG }, 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "ds-idrepo-1" 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG }, 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "value": [ 09:40:24 DEBUG "pod": "ds-cts-1" 09:40:24 DEBUG [...] 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG }, 09:40:24 DEBUG 09:40:24 DEBUG "0" 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG ] 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG 09:40:24 DEBUG } 09:40:24 DEBUG "0" 09:40:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:40:24 DEBUG ] 09:40:24 DEBUG ] 09:40:24 DEBUG }, 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 DEBUG "value": [ 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG } 09:40:24 DEBUG "4678852608" 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263224" 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "ds-idrepo-2" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0.014082998607888648" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "instance": "10.142.15.199:9100" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "instance": "10.142.15.212:9100" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "31543.733333333334" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263224" 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "pod": "ds-idrepo-0" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "ds-idrepo-1" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG { 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG "data": { 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:24 DEBUG "result": [ 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:40:24 DEBUG "data": { 09:40:24 DEBUG }, 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "value": [ 09:40:24 DEBUG "result": [ 09:40:24 DEBUG 09:40:24 DEBUG { 09:40:24 DEBUG 09:40:24 DEBUG "metric": { 09:40:24 DEBUG 09:40:24 DEBUG "instance": "10.142.15.199:9100" 09:40:24 DEBUG [...] 09:40:24 DEBUG }, 09:40:24 DEBUG 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "ds-idrepo-2" 09:40:24 DEBUG 09:40:24 DEBUG }, 09:40:24 DEBUG [...] 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG 09:40:24 DEBUG "1315.8665000892381" 09:40:24 DEBUG 09:40:24 DEBUG ] 09:40:24 DEBUG "instance": "10.142.15.212:9100" 09:40:24 DEBUG } 09:40:24 DEBUG }, 09:40:24 DEBUG ] 09:40:24 DEBUG "value": [ 09:40:24 DEBUG } 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG } 09:40:24 DEBUG "0.0003333333333406093" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "pod": "ds-cts-2" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "ds-cts-1" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0.2" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "pod": "ds-idrepo-0" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "ds-cts-1" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "instance": "10.142.15.199:9100" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "instance": "10.142.15.212:9100" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0.002799999999999632" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG "data": { 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG "result": [ 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:24 DEBUG { 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG "metric": { 09:40:24 DEBUG { 09:40:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG }, 09:40:24 DEBUG "data": { 09:40:24 DEBUG "value": [ 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG 09:40:24 DEBUG "result": [ 09:40:24 DEBUG 09:40:24 DEBUG { 09:40:24 DEBUG 09:40:24 DEBUG "metric": { 09:40:24 DEBUG [...] 09:40:24 DEBUG "instance": "10.142.15.199:9100" 09:40:24 DEBUG 09:40:24 DEBUG }, 09:40:24 DEBUG 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:40:24 DEBUG 09:40:24 DEBUG }, 09:40:24 DEBUG 09:40:24 DEBUG "value": [ 09:40:24 DEBUG [...] 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG 09:40:24 DEBUG "0" 09:40:24 DEBUG 09:40:24 DEBUG ] 09:40:24 DEBUG 09:40:24 DEBUG } 09:40:24 DEBUG "instance": "10.142.15.212:9100" 09:40:24 DEBUG ] 09:40:24 DEBUG }, 09:40:24 DEBUG } 09:40:24 DEBUG "value": [ 09:40:24 DEBUG } 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "28896.86666666667" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "pod": "ds-idrepo-2" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "0" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263224" 09:40:24 INFO [http_cmd]: http status code OK 09:40:24 DEBUG --- status code --- 09:40:24 DEBUG http status code is 200 (expected 200) 09:40:24 DEBUG --- http response --- 09:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:40:24 DEBUG ----- output ----- 09:40:24 DEBUG { 09:40:24 DEBUG "status": "success", 09:40:24 DEBUG "data": { 09:40:24 DEBUG "resultType": "vector", 09:40:24 DEBUG "result": [ 09:40:24 DEBUG { 09:40:24 DEBUG "metric": { 09:40:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG [...] 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG 09:40:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:40:24 DEBUG }, 09:40:24 DEBUG "value": [ 09:40:24 DEBUG 1685263224, 09:40:24 DEBUG "3.4173591662425564" 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG ] 09:40:24 DEBUG } 09:40:24 DEBUG } 09:40:31 INFO 09:40:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:40:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:40:31 INFO [loop_until]: OK (rc = 0) 09:40:31 DEBUG --- stdout --- 09:40:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 5m 5815Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 13m 456Mi ds-cts-1 9m 409Mi ds-cts-2 9m 426Mi ds-idrepo-0 6737m 9897Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10061m 4469Mi idm-58dc667486-h86fb 8904m 4110Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3421m 2675Mi 09:40:31 DEBUG --- stderr --- 09:40:31 DEBUG 09:40:32 INFO 09:40:32 INFO [loop_until]: kubectl --namespace=xlou top node 09:40:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:40:33 INFO [loop_until]: OK (rc = 0) 09:40:33 DEBUG --- stdout --- 09:40:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 95m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 65m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 130m 0% 1534Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 82m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 13758m 86% 5650Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 8849m 55% 5249Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 82m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1092Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 63m 0% 1102Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1071Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13161Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6909m 43% 10436Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14328Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3511m 22% 3615Mi 6% 09:40:33 DEBUG --- stderr --- 09:40:33 DEBUG 09:41:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263283" 09:41:23 INFO [http_cmd]: http status code OK 09:41:23 DEBUG --- status code --- 09:41:23 DEBUG http status code is 200 (expected 200) 09:41:23 DEBUG --- http response --- 09:41:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:41:23 DEBUG ----- output ----- 09:41:23 DEBUG { 09:41:23 DEBUG "status": "success", 09:41:23 DEBUG "data": { 09:41:23 DEBUG "resultType": "vector", 09:41:23 DEBUG "result": [ 09:41:23 DEBUG { 09:41:23 DEBUG "metric": { 09:41:23 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:41:23 DEBUG }, 09:41:23 DEBUG "value": [ 09:41:23 DEBUG 09:41:23 DEBUG 09:41:23 DEBUG 09:41:23 DEBUG [...] 09:41:23 DEBUG 09:41:23 DEBUG 09:41:23 DEBUG 09:41:23 DEBUG "pod": "idm-58dc667486-gclr8" 09:41:23 DEBUG }, 09:41:23 DEBUG "value": [ 09:41:23 DEBUG 1685263283, 09:41:23 DEBUG "0" 09:41:23 DEBUG ] 09:41:23 DEBUG } 09:41:23 DEBUG ] 09:41:23 DEBUG } 09:41:23 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "ds-idrepo-0" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "ds-idrepo-1" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "ds-idrepo-0" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [...] 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:41:24 DEBUG 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG 09:41:24 DEBUG { 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG "pod": "ds-idrepo-1" 09:41:24 DEBUG "data": { 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG }, 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG "value": [ 09:41:24 DEBUG "result": [ 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG { 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG "0" 09:41:24 DEBUG "metric": { 09:41:24 DEBUG { 09:41:24 DEBUG ] 09:41:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG } 09:41:24 DEBUG }, 09:41:24 DEBUG "data": { 09:41:24 DEBUG ] 09:41:24 DEBUG "value": [ 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG } 09:41:24 DEBUG 09:41:24 DEBUG "result": [ 09:41:24 DEBUG } 09:41:24 DEBUG 09:41:24 DEBUG { 09:41:24 DEBUG 09:41:24 DEBUG "metric": { 09:41:24 DEBUG [...] 09:41:24 DEBUG "pod": "ds-idrepo-0" 09:41:24 DEBUG 09:41:24 DEBUG }, 09:41:24 DEBUG 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:41:24 DEBUG 09:41:24 DEBUG }, 09:41:24 DEBUG 09:41:24 DEBUG "value": [ 09:41:24 DEBUG [...] 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG 09:41:24 DEBUG "4690333696" 09:41:24 DEBUG 09:41:24 DEBUG ] 09:41:24 DEBUG 09:41:24 DEBUG } 09:41:24 DEBUG "pod": "ds-cts-1" 09:41:24 DEBUG ] 09:41:24 DEBUG }, 09:41:24 DEBUG } 09:41:24 DEBUG "value": [ 09:41:24 DEBUG } 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "ds-idrepo-0" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "ds-idrepo-1" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "ds-cts-2" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "instance": "10.142.15.199:9100" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "instance": "10.142.15.212:9100" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "31410.366666666665" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263284" 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 DEBUG 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG 09:41:24 DEBUG { 09:41:24 DEBUG "pod": "ds-cts-2" 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG }, 09:41:24 DEBUG "data": { 09:41:24 DEBUG "value": [ 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "result": [ 09:41:24 DEBUG "0" 09:41:24 DEBUG { 09:41:24 DEBUG ] 09:41:24 DEBUG "metric": { 09:41:24 DEBUG } 09:41:24 DEBUG "instance": "10.142.15.199:9100" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG ] 09:41:24 DEBUG }, 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG } 09:41:24 DEBUG "value": [ 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG } 09:41:24 DEBUG 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG 09:41:24 DEBUG { 09:41:24 DEBUG [...] 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG 09:41:24 DEBUG "data": { 09:41:24 DEBUG 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG 09:41:24 DEBUG "result": [ 09:41:24 DEBUG "instance": "10.142.15.212:9100" 09:41:24 DEBUG { 09:41:24 DEBUG }, 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "value": [ 09:41:24 DEBUG "pod": "ds-cts-2" 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG }, 09:41:24 DEBUG "0" 09:41:24 DEBUG "value": [ 09:41:24 DEBUG ] 09:41:24 DEBUG 09:41:24 DEBUG } 09:41:24 DEBUG 09:41:24 DEBUG ] 09:41:24 DEBUG 09:41:24 DEBUG } 09:41:24 DEBUG [...] 09:41:24 DEBUG } 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "ds-cts-1" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "ds-idrepo-0" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "ds-cts-1" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0.2" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "instance": "10.142.15.199:9100" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "instance": "10.142.15.212:9100" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0.0030333333333331324" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG { 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG "data": { 09:41:24 DEBUG { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG "data": { 09:41:24 DEBUG { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "result": [ 09:41:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:41:24 DEBUG { 09:41:24 DEBUG }, 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "value": [ 09:41:24 DEBUG "instance": "10.142.15.199:9100" 09:41:24 DEBUG 09:41:24 DEBUG }, 09:41:24 DEBUG 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:41:24 DEBUG 09:41:24 DEBUG }, 09:41:24 DEBUG 09:41:24 DEBUG "value": [ 09:41:24 DEBUG "instance": "10.142.15.212:9100" 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG }, 09:41:24 DEBUG "0" 09:41:24 DEBUG "value": [ 09:41:24 DEBUG ] 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG } 09:41:24 DEBUG "28921.966666666667" 09:41:24 DEBUG ] 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "0" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263284" 09:41:24 INFO [http_cmd]: http status code OK 09:41:24 DEBUG --- status code --- 09:41:24 DEBUG http status code is 200 (expected 200) 09:41:24 DEBUG --- http response --- 09:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:41:24 DEBUG ----- output ----- 09:41:24 DEBUG { 09:41:24 DEBUG "status": "success", 09:41:24 DEBUG "data": { 09:41:24 DEBUG "resultType": "vector", 09:41:24 DEBUG "result": [ 09:41:24 DEBUG { 09:41:24 DEBUG "metric": { 09:41:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG [...] 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG 09:41:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:41:24 DEBUG }, 09:41:24 DEBUG "value": [ 09:41:24 DEBUG 1685263284, 09:41:24 DEBUG "3.404524507941359" 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG ] 09:41:24 DEBUG } 09:41:24 DEBUG } 09:41:31 INFO 09:41:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:41:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:41:31 INFO [loop_until]: OK (rc = 0) 09:41:31 DEBUG --- stdout --- 09:41:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5742Mi am-7849cf7bdb-cwb58 5m 5815Mi am-7849cf7bdb-vcl2f 4m 5768Mi ds-cts-0 8m 455Mi ds-cts-1 9m 409Mi ds-cts-2 8m 426Mi ds-idrepo-0 7071m 9896Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10007m 4476Mi idm-58dc667486-h86fb 9025m 4116Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3368m 2675Mi 09:41:31 DEBUG --- stderr --- 09:41:31 DEBUG 09:41:33 INFO 09:41:33 INFO [loop_until]: kubectl --namespace=xlou top node 09:41:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:41:33 INFO [loop_until]: OK (rc = 0) 09:41:33 DEBUG --- stdout --- 09:41:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 98m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 67m 0% 6652Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 123m 0% 1547Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 80m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14421m 90% 5658Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9340m 58% 5254Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 84m 0% 1349Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1092Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1097Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1073Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13161Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6804m 42% 10433Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14329Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3519m 22% 3614Mi 6% 09:41:33 DEBUG --- stderr --- 09:41:33 DEBUG 09:42:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263343" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263343, 09:42:24 DEBUG "0" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:42:24 DEBUG "pod": "ds-idrepo-0" 09:42:24 DEBUG }, 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:42:24 DEBUG "pod": "ds-idrepo-1" 09:42:24 DEBUG }, 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0" 09:42:24 DEBUG "0" 09:42:24 DEBUG ] 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "ds-idrepo-0" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "ds-cts-1" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "4700114944" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "ds-idrepo-0" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "ds-idrepo-1" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "ds-idrepo-0" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "ds-idrepo-1" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263344" 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG { 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG "data": { 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "data": { 09:42:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:42:24 DEBUG "resultType": "vector", 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG }, 09:42:24 DEBUG "result": [ 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG "value": [ 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG { 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG "metric": { 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:42:24 DEBUG 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG }, 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG [...] 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:42:24 DEBUG "value": [ 09:42:24 DEBUG { 09:42:24 DEBUG 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG 09:42:24 DEBUG { 09:42:24 DEBUG 09:42:24 DEBUG "data": { 09:42:24 DEBUG 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "pod": "idm-58dc667486-gclr8" 09:42:24 DEBUG "data": { 09:42:24 DEBUG [...] 09:42:24 DEBUG "result": [ 09:42:24 DEBUG }, 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG 09:42:24 DEBUG { 09:42:24 DEBUG "value": [ 09:42:24 DEBUG "result": [ 09:42:24 DEBUG 09:42:24 DEBUG "metric": { 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG { 09:42:24 DEBUG 09:42:24 DEBUG "pod": "ds-idrepo-0" 09:42:24 DEBUG "6.233587736566985" 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:42:24 DEBUG }, 09:42:24 DEBUG ] 09:42:24 DEBUG "instance": "10.142.15.199:9100" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG } 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG ] 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG 09:42:24 DEBUG } 09:42:24 DEBUG 09:42:24 DEBUG "0" 09:42:24 DEBUG 09:42:24 DEBUG } 09:42:24 DEBUG 09:42:24 DEBUG ] 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG } 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG ] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG } 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG } 09:42:24 DEBUG "pod": "ds-cts-1" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG 09:42:24 DEBUG }, 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG "instance": "10.142.15.212:9100" 09:42:24 DEBUG "value": [ 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG }, 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG "value": [ 09:42:24 DEBUG "0" 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG ] 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG "31431.93333333333" 09:42:24 DEBUG } 09:42:24 DEBUG { 09:42:24 DEBUG ] 09:42:24 DEBUG ] 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 DEBUG "data": { 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG } 09:42:24 DEBUG "result": [ 09:42:24 DEBUG } 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "idm-58dc667486-gclr8" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "96263.10481251846" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 DEBUG "resultType": "vector", 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263344" 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0.2" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "ds-cts-2" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "ds-cts-1" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "instance": "10.142.15.199:9100" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "instance": "10.142.15.212:9100" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "instance": "10.142.15.199:9100" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "instance": "10.142.15.212:9100" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0.0026000000000003355" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263344" 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:42:24 DEBUG { 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG { 09:42:24 DEBUG "data": { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "result": [ 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG { 09:42:24 DEBUG "result": [ 09:42:24 DEBUG "metric": { 09:42:24 DEBUG { 09:42:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:42:24 DEBUG "metric": { 09:42:24 DEBUG }, 09:42:24 DEBUG "instance": "10.142.15.199:9100" 09:42:24 DEBUG "value": [ 09:42:24 DEBUG }, 09:42:24 DEBUG 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:42:24 DEBUG 09:42:24 DEBUG }, 09:42:24 DEBUG "instance": "10.142.15.212:9100" 09:42:24 DEBUG "value": [ 09:42:24 DEBUG }, 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG "0" 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG ] 09:42:24 DEBUG "28490.9" 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "pod": "idm-58dc667486-gclr8" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "0" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263344" 09:42:24 INFO [http_cmd]: http status code OK 09:42:24 DEBUG --- status code --- 09:42:24 DEBUG http status code is 200 (expected 200) 09:42:24 DEBUG --- http response --- 09:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:42:24 DEBUG ----- output ----- 09:42:24 DEBUG { 09:42:24 DEBUG "status": "success", 09:42:24 DEBUG "data": { 09:42:24 DEBUG "resultType": "vector", 09:42:24 DEBUG "result": [ 09:42:24 DEBUG { 09:42:24 DEBUG "metric": { 09:42:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG [...] 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG 09:42:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:42:24 DEBUG }, 09:42:24 DEBUG "value": [ 09:42:24 DEBUG 1685263344, 09:42:24 DEBUG "3.5178813415057206" 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG ] 09:42:24 DEBUG } 09:42:24 DEBUG } 09:42:31 INFO 09:42:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:42:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:42:31 INFO [loop_until]: OK (rc = 0) 09:42:31 DEBUG --- stdout --- 09:42:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 5m 5815Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 8m 455Mi ds-cts-1 13m 409Mi ds-cts-2 8m 426Mi ds-idrepo-0 6840m 9897Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 13m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10381m 4485Mi idm-58dc667486-h86fb 9245m 4126Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3356m 2680Mi 09:42:31 DEBUG --- stderr --- 09:42:31 DEBUG 09:42:33 INFO 09:42:33 INFO [loop_until]: kubectl --namespace=xlou top node 09:42:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:42:33 INFO [loop_until]: OK (rc = 0) 09:42:33 DEBUG --- stdout --- 09:42:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 90m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 68m 0% 6652Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 118m 0% 1546Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 81m 0% 6818Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14610m 91% 5667Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9441m 59% 5264Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1093Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1101Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 65m 0% 1074Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13164Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7069m 44% 10437Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 64m 0% 14331Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3405m 21% 3618Mi 6% 09:42:33 DEBUG --- stderr --- 09:42:33 DEBUG 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-idrepo-2" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0.6097213496949645" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "ds-idrepo-0" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-idrepo-1" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "ds-idrepo-0" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-cts-1" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263404" 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG { 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG "data": { 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "data": { 09:43:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG }, 09:43:24 DEBUG "result": [ 09:43:24 DEBUG "value": [ 09:43:24 DEBUG { 09:43:24 DEBUG 09:43:24 DEBUG "metric": { 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-idrepo-0" 09:43:24 DEBUG 09:43:24 DEBUG }, 09:43:24 DEBUG [...] 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:43:24 DEBUG [...] 09:43:24 DEBUG }, 09:43:24 DEBUG 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG 09:43:24 DEBUG "4709031936" 09:43:24 DEBUG "pod": "ds-idrepo-1" 09:43:24 DEBUG ] 09:43:24 DEBUG }, 09:43:24 DEBUG } 09:43:24 DEBUG "value": [ 09:43:24 DEBUG ] 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG } 09:43:24 DEBUG "0" 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "ds-idrepo-0" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-idrepo-1" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263404" 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263404" 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "ds-idrepo-0" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-cts-1" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG { 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:43:24 DEBUG "data": { 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG { 09:43:24 DEBUG "result": [ 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG { 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG "data": { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG "result": [ 09:43:24 DEBUG }, 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:43:24 DEBUG { 09:43:24 DEBUG "value": [ 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG "metric": { 09:43:24 DEBUG 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG { 09:43:24 DEBUG "instance": "10.142.15.199:9100" 09:43:24 DEBUG 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG }, 09:43:24 DEBUG 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG "data": { 09:43:24 DEBUG "value": [ 09:43:24 DEBUG [...] 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG "result": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG { 09:43:24 DEBUG { 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "metric": { 09:43:24 DEBUG [...] 09:43:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG "data": { 09:43:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:43:24 DEBUG 09:43:24 DEBUG }, 09:43:24 DEBUG { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG }, 09:43:24 DEBUG 09:43:24 DEBUG "value": [ 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "data": { 09:43:24 DEBUG { 09:43:24 DEBUG 09:43:24 DEBUG "instance": "10.142.15.212:9100" 09:43:24 DEBUG "0" 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "metric": { 09:43:24 DEBUG 09:43:24 DEBUG }, 09:43:24 DEBUG ] 09:43:24 DEBUG "result": [ 09:43:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:43:24 DEBUG 09:43:24 DEBUG "value": [ 09:43:24 DEBUG } 09:43:24 DEBUG { 09:43:24 DEBUG }, 09:43:24 DEBUG [...] 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG ] 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG "31428.199999999997" 09:43:24 DEBUG } 09:43:24 DEBUG "instance": "10.142.15.199:9100" 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG }, 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG } 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-idrepo-2" 09:43:24 DEBUG ] 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG }, 09:43:24 DEBUG } 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "value": [ 09:43:24 DEBUG } 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG "0.009406887933746689" 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-idrepo-2" 09:43:24 DEBUG ] 09:43:24 DEBUG 09:43:24 DEBUG }, 09:43:24 DEBUG } 09:43:24 DEBUG 09:43:24 DEBUG "value": [ 09:43:24 DEBUG ] 09:43:24 DEBUG "instance": "10.142.15.212:9100" 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG } 09:43:24 DEBUG }, 09:43:24 DEBUG "389152541.5852243" 09:43:24 DEBUG } 09:43:24 DEBUG "value": [ 09:43:24 DEBUG ] 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG } 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0.2" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG } 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG } 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "ds-cts-2" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-cts-1" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "instance": "10.142.15.199:9100" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "instance": "10.142.15.212:9100" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0.0028666666666666177" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "instance": "10.142.15.199:9100" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "instance": "10.142.15.212:9100" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "30385.533333333333" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263404" 09:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263404" 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "pod": "ds-idrepo-2" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "0" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:24 INFO [http_cmd]: http status code OK 09:43:24 DEBUG --- status code --- 09:43:24 DEBUG http status code is 200 (expected 200) 09:43:24 DEBUG --- http response --- 09:43:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:43:24 DEBUG ----- output ----- 09:43:24 DEBUG { 09:43:24 DEBUG "status": "success", 09:43:24 DEBUG "data": { 09:43:24 DEBUG "resultType": "vector", 09:43:24 DEBUG "result": [ 09:43:24 DEBUG { 09:43:24 DEBUG "metric": { 09:43:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG [...] 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG 09:43:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:43:24 DEBUG }, 09:43:24 DEBUG "value": [ 09:43:24 DEBUG 1685263404, 09:43:24 DEBUG "3.4115524145844933" 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG ] 09:43:24 DEBUG } 09:43:24 DEBUG } 09:43:31 INFO 09:43:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:43:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:43:31 INFO [loop_until]: OK (rc = 0) 09:43:31 DEBUG --- stdout --- 09:43:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 5m 5815Mi am-7849cf7bdb-vcl2f 7m 5768Mi ds-cts-0 9m 455Mi ds-cts-1 9m 409Mi ds-cts-2 15m 425Mi ds-idrepo-0 6868m 9897Mi ds-idrepo-1 15m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 9997m 4492Mi idm-58dc667486-h86fb 9012m 4132Mi lodemon-cfd7686d9-6dlt4 1m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3506m 2680Mi 09:43:31 DEBUG --- stderr --- 09:43:31 DEBUG 09:43:33 INFO 09:43:33 INFO [loop_until]: kubectl --namespace=xlou top node 09:43:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:43:33 INFO [loop_until]: OK (rc = 0) 09:43:33 DEBUG --- stdout --- 09:43:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 90m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 72m 0% 6656Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 132m 0% 1546Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6815Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14037m 88% 5672Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9270m 58% 5272Mi 8% gke-xlou-cdm-default-pool-f05840a3-bqd0 85m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 64m 0% 1091Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1100Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 64m 0% 1074Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13165Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7065m 44% 10435Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14330Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3551m 22% 3621Mi 6% 09:43:33 DEBUG --- stderr --- 09:43:33 DEBUG 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "ds-idrepo-2" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0.673120598828146" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "ds-idrepo-0" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "ds-idrepo-1" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "ds-idrepo-0" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "ds-cts-1" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263464" 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "4804071424" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:44:24 DEBUG { 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG { 09:44:24 DEBUG "data": { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "result": [ 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG { 09:44:24 DEBUG "result": [ 09:44:24 DEBUG "metric": { 09:44:24 DEBUG { 09:44:24 DEBUG "pod": "ds-idrepo-0" 09:44:24 DEBUG "metric": { 09:44:24 DEBUG }, 09:44:24 DEBUG "pod": "ds-idrepo-0" 09:44:24 DEBUG "value": [ 09:44:24 DEBUG }, 09:44:24 DEBUG 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "ds-idrepo-1" 09:44:24 DEBUG 09:44:24 DEBUG }, 09:44:24 DEBUG "pod": "ds-idrepo-1" 09:44:24 DEBUG "value": [ 09:44:24 DEBUG }, 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG "0" 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG ] 09:44:24 DEBUG "0" 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "ds-idrepo-0" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "ds-cts-1" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263464" 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG { 09:44:24 DEBUG { 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "data": { 09:44:24 DEBUG "data": { 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG "result": [ 09:44:24 DEBUG "result": [ 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG { 09:44:24 DEBUG { 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:44:24 DEBUG "pod": "ds-cts-2" 09:44:24 DEBUG "instance": "10.142.15.199:9100" 09:44:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:44:24 DEBUG }, 09:44:24 DEBUG "instance": "10.142.15.199:9100" 09:44:24 DEBUG }, 09:44:24 DEBUG }, 09:44:24 DEBUG }, 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG "value": [ 09:44:24 DEBUG "value": [ 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG [...] 09:44:24 DEBUG [...] 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:44:24 DEBUG 09:44:24 DEBUG "pod": "ds-idrepo-2" 09:44:24 DEBUG "pod": "ds-cts-1" 09:44:24 DEBUG "instance": "10.142.15.212:9100" 09:44:24 DEBUG "pod": "ds-idrepo-2" 09:44:24 DEBUG }, 09:44:24 DEBUG "instance": "10.142.15.212:9100" 09:44:24 DEBUG }, 09:44:24 DEBUG }, 09:44:24 DEBUG }, 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG "value": [ 09:44:24 DEBUG "value": [ 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0" 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "410944490.6838682" 09:44:24 DEBUG "0" 09:44:24 DEBUG "31358.93333333333" 09:44:24 DEBUG "0.009187032731276168" 09:44:24 DEBUG ] 09:44:24 DEBUG "0.0006666666666584812" 09:44:24 DEBUG ] 09:44:24 DEBUG ] 09:44:24 DEBUG ] 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG ] 09:44:24 DEBUG ] 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0.2" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "instance": "10.142.15.199:9100" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "instance": "10.142.15.212:9100" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0.00290000000000011" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263464" 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "instance": "10.142.15.199:9100" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "instance": "10.142.15.212:9100" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "28563.166666666664" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "3.4387311497951316" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263464" 09:44:24 INFO [http_cmd]: http status code OK 09:44:24 DEBUG --- status code --- 09:44:24 DEBUG http status code is 200 (expected 200) 09:44:24 DEBUG --- http response --- 09:44:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:44:24 DEBUG ----- output ----- 09:44:24 DEBUG { 09:44:24 DEBUG "status": "success", 09:44:24 DEBUG "data": { 09:44:24 DEBUG "resultType": "vector", 09:44:24 DEBUG "result": [ 09:44:24 DEBUG { 09:44:24 DEBUG "metric": { 09:44:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG [...] 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG 09:44:24 DEBUG "pod": "ds-idrepo-2" 09:44:24 DEBUG }, 09:44:24 DEBUG "value": [ 09:44:24 DEBUG 1685263464, 09:44:24 DEBUG "0" 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG ] 09:44:24 DEBUG } 09:44:24 DEBUG } 09:44:31 INFO 09:44:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:44:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:44:31 INFO [loop_until]: OK (rc = 0) 09:44:31 DEBUG --- stdout --- 09:44:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 5m 5815Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 8m 455Mi ds-cts-1 8m 409Mi ds-cts-2 8m 425Mi ds-idrepo-0 6884m 9897Mi ds-idrepo-1 15m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10193m 4500Mi idm-58dc667486-h86fb 9164m 4150Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3445m 2681Mi 09:44:31 DEBUG --- stderr --- 09:44:31 DEBUG 09:44:33 INFO 09:44:33 INFO [loop_until]: kubectl --namespace=xlou top node 09:44:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:44:33 INFO [loop_until]: OK (rc = 0) 09:44:33 DEBUG --- stdout --- 09:44:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 92m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 65m 0% 6655Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 127m 0% 1546Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14615m 91% 5680Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9323m 58% 5281Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 82m 0% 1346Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1090Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1099Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13166Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7031m 44% 10434Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 68m 0% 14332Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3536m 22% 3624Mi 6% 09:44:33 DEBUG --- stderr --- 09:44:33 DEBUG 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "ds-idrepo-2" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "0.9381188978840206" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "pod": "ds-idrepo-0" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "ds-idrepo-1" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "0" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:45:24 DEBUG { 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG { 09:45:24 DEBUG { 09:45:24 DEBUG "data": { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "data": { 09:45:24 DEBUG "result": [ 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG { 09:45:24 DEBUG "result": [ 09:45:24 DEBUG "result": [ 09:45:24 DEBUG "metric": { 09:45:24 DEBUG { 09:45:24 DEBUG { 09:45:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG }, 09:45:24 DEBUG "pod": "ds-idrepo-0" 09:45:24 DEBUG "pod": "ds-idrepo-0" 09:45:24 DEBUG "value": [ 09:45:24 DEBUG }, 09:45:24 DEBUG }, 09:45:24 DEBUG 09:45:24 DEBUG "value": [ 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG }, 09:45:24 DEBUG "pod": "ds-idrepo-1" 09:45:24 DEBUG "pod": "ds-cts-1" 09:45:24 DEBUG "value": [ 09:45:24 DEBUG }, 09:45:24 DEBUG }, 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG "value": [ 09:45:24 DEBUG "0" 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG ] 09:45:24 DEBUG "0" 09:45:24 DEBUG "0" 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG ] 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263524" 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "4730695680" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "pod": "ds-idrepo-0" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "ds-idrepo-1" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "0" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "pod": "ds-idrepo-0" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "ds-cts-1" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "0" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:45:24 DEBUG "instance": "10.142.15.199:9100" 09:45:24 DEBUG }, 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG 09:45:24 DEBUG "pod": "ds-idrepo-2" 09:45:24 DEBUG { 09:45:24 DEBUG "instance": "10.142.15.212:9100" 09:45:24 DEBUG }, 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG "data": { 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "6688.857232201799" 09:45:24 DEBUG "result": [ 09:45:24 DEBUG "31432.033333333333" 09:45:24 DEBUG ] 09:45:24 DEBUG { 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG "metric": { 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG }, 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 DEBUG "value": [ 09:45:24 DEBUG } 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "ds-idrepo-2" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "0.014846090511081576" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "0" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263524" 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "pod": "ds-cts-2" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "ds-cts-1" 09:45:24 DEBUG }, 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG "value": [ 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG "0" 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG ] 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:45:24 DEBUG } 09:45:24 DEBUG ----- output ----- 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG ] 09:45:24 DEBUG { 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG } 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG } 09:45:24 DEBUG "data": { 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:45:24 DEBUG "result": [ 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "instance": "10.142.15.199:9100" 09:45:24 DEBUG "data": { 09:45:24 DEBUG }, 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "value": [ 09:45:24 DEBUG "result": [ 09:45:24 DEBUG 09:45:24 DEBUG { 09:45:24 DEBUG 09:45:24 DEBUG "metric": { 09:45:24 DEBUG 09:45:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:45:24 DEBUG [...] 09:45:24 DEBUG }, 09:45:24 DEBUG 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "instance": "10.142.15.212:9100" 09:45:24 DEBUG 09:45:24 DEBUG }, 09:45:24 DEBUG [...] 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG 09:45:24 DEBUG "0" 09:45:24 DEBUG 09:45:24 DEBUG ] 09:45:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:45:24 DEBUG } 09:45:24 DEBUG }, 09:45:24 DEBUG ] 09:45:24 DEBUG "value": [ 09:45:24 DEBUG } 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG } 09:45:24 DEBUG "0.2" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG { 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG "status": "success", 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263524" 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:45:24 DEBUG "data": { 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG { 09:45:24 DEBUG "result": [ 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG { 09:45:24 DEBUG "data": { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:45:24 DEBUG "result": [ 09:45:24 DEBUG }, 09:45:24 DEBUG { 09:45:24 DEBUG "value": [ 09:45:24 DEBUG "metric": { 09:45:24 DEBUG 09:45:24 DEBUG "instance": "10.142.15.199:9100" 09:45:24 DEBUG 09:45:24 DEBUG }, 09:45:24 DEBUG 09:45:24 DEBUG "value": [ 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:45:24 DEBUG 09:45:24 DEBUG }, 09:45:24 DEBUG 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "instance": "10.142.15.212:9100" 09:45:24 DEBUG "0" 09:45:24 DEBUG }, 09:45:24 DEBUG ] 09:45:24 DEBUG "value": [ 09:45:24 DEBUG } 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG ] 09:45:24 DEBUG "28809.3" 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "instance": "10.142.15.199:9100" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "instance": "10.142.15.212:9100" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "0.0025333333333331133" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "3.4167946829021703" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "0" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263524" 09:45:24 INFO [http_cmd]: http status code OK 09:45:24 DEBUG --- status code --- 09:45:24 DEBUG http status code is 200 (expected 200) 09:45:24 DEBUG --- http response --- 09:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:45:24 DEBUG ----- output ----- 09:45:24 DEBUG { 09:45:24 DEBUG "status": "success", 09:45:24 DEBUG "data": { 09:45:24 DEBUG "resultType": "vector", 09:45:24 DEBUG "result": [ 09:45:24 DEBUG { 09:45:24 DEBUG "metric": { 09:45:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG [...] 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG 09:45:24 DEBUG "pod": "ds-idrepo-2" 09:45:24 DEBUG }, 09:45:24 DEBUG "value": [ 09:45:24 DEBUG 1685263524, 09:45:24 DEBUG "0" 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG ] 09:45:24 DEBUG } 09:45:24 DEBUG } 09:45:31 INFO 09:45:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:45:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:45:31 INFO [loop_until]: OK (rc = 0) 09:45:31 DEBUG --- stdout --- 09:45:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 5m 5815Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 9m 456Mi ds-cts-1 10m 409Mi ds-cts-2 9m 424Mi ds-idrepo-0 7189m 9897Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10914m 4508Mi idm-58dc667486-h86fb 9024m 4157Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3366m 2684Mi 09:45:31 DEBUG --- stderr --- 09:45:31 DEBUG 09:45:33 INFO 09:45:33 INFO [loop_until]: kubectl --namespace=xlou top node 09:45:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:45:33 INFO [loop_until]: OK (rc = 0) 09:45:33 DEBUG --- stdout --- 09:45:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 91m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 68m 0% 6657Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 118m 0% 1546Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15166m 95% 5687Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9281m 58% 5289Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 81m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1090Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1104Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1073Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13166Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7378m 46% 10437Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14331Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3542m 22% 3624Mi 6% 09:45:33 DEBUG --- stderr --- 09:45:33 DEBUG 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "idm-58dc667486-gclr8" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "ds-idrepo-0" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "ds-idrepo-1" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "ds-idrepo-0" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "ds-cts-1" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:46:24 DEBUG "pod": "ds-idrepo-0" 09:46:24 DEBUG }, 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:46:24 DEBUG "pod": "ds-idrepo-1" 09:46:24 DEBUG }, 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0.06666666666666667" 09:46:24 DEBUG "0" 09:46:24 DEBUG ] 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263584" 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "ds-idrepo-0" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG [...] 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:46:24 DEBUG 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG "pod": "ds-idrepo-1" 09:46:24 DEBUG { 09:46:24 DEBUG }, 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "value": [ 09:46:24 DEBUG "data": { 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "0" 09:46:24 DEBUG "result": [ 09:46:24 DEBUG ] 09:46:24 DEBUG { 09:46:24 DEBUG } 09:46:24 DEBUG "metric": { 09:46:24 DEBUG ] 09:46:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:46:24 DEBUG } 09:46:24 DEBUG }, 09:46:24 DEBUG } 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "4740202496" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "ds-idrepo-0" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "ds-cts-1" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263584" 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "idm-58dc667486-gclr8" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "4.66419134072859" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG { 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "data": { 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG "result": [ 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG { 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "instance": "10.142.15.199:9100" 09:46:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:46:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:46:24 DEBUG }, 09:46:24 DEBUG }, 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG "value": [ 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG [...] 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "instance": "10.142.15.212:9100" 09:46:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:46:24 DEBUG "pod": "idm-58dc667486-gclr8" 09:46:24 DEBUG }, 09:46:24 DEBUG }, 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG "value": [ 09:46:24 DEBUG "value": [ 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263584" 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "31378.333333333332" 09:46:24 DEBUG "0.06666666666666667" 09:46:24 DEBUG "45704.080219418545" 09:46:24 DEBUG ] 09:46:24 DEBUG ] 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG ] 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "ds-cts-2" 09:46:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:46:24 DEBUG }, 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "ds-cts-1" 09:46:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:46:24 DEBUG }, 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0" 09:46:24 DEBUG "0.2" 09:46:24 DEBUG ] 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG } 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "instance": "10.142.15.199:9100" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "instance": "10.142.15.212:9100" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "28471.633333333335" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "instance": "10.142.15.199:9100" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "instance": "10.142.15.212:9100" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0.0003333333333330302" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "instance": "10.142.15.199:9100" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "instance": "10.142.15.212:9100" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0.002466666666666602" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "3.4256360087406073" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263584" 09:46:24 INFO [http_cmd]: http status code OK 09:46:24 DEBUG --- status code --- 09:46:24 DEBUG http status code is 200 (expected 200) 09:46:24 DEBUG --- http response --- 09:46:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:46:24 DEBUG ----- output ----- 09:46:24 DEBUG { 09:46:24 DEBUG "status": "success", 09:46:24 DEBUG "data": { 09:46:24 DEBUG "resultType": "vector", 09:46:24 DEBUG "result": [ 09:46:24 DEBUG { 09:46:24 DEBUG "metric": { 09:46:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG [...] 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG 09:46:24 DEBUG "pod": "idm-58dc667486-gclr8" 09:46:24 DEBUG }, 09:46:24 DEBUG "value": [ 09:46:24 DEBUG 1685263584, 09:46:24 DEBUG "0" 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG ] 09:46:24 DEBUG } 09:46:24 DEBUG } 09:46:31 INFO 09:46:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:46:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:46:31 INFO [loop_until]: OK (rc = 0) 09:46:31 DEBUG --- stdout --- 09:46:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 5m 5816Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 9m 455Mi ds-cts-1 8m 409Mi ds-cts-2 9m 424Mi ds-idrepo-0 6974m 9897Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 9769m 4517Mi idm-58dc667486-h86fb 8736m 4166Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3420m 2684Mi 09:46:31 DEBUG --- stderr --- 09:46:31 DEBUG 09:46:33 INFO 09:46:33 INFO [loop_until]: kubectl --namespace=xlou top node 09:46:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:46:33 INFO [loop_until]: OK (rc = 0) 09:46:33 DEBUG --- stdout --- 09:46:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 87m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6653Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 128m 0% 1546Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6817Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14709m 92% 5699Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9148m 57% 5294Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 82m 0% 1344Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1089Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1103Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1074Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13166Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7106m 44% 10435Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14332Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3494m 21% 3620Mi 6% 09:46:33 DEBUG --- stderr --- 09:46:33 DEBUG 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-idrepo-2" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0.9717886801342835" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "ds-idrepo-0" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-idrepo-1" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "ds-idrepo-0" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-cts-1" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "ds-idrepo-0" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-idrepo-1" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "4745682944" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "ds-idrepo-0" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-cts-1" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "ds-idrepo-0" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-idrepo-1" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "ds-cts-2" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-cts-1" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263644" 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG "data": { 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG "result": [ 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:47:24 DEBUG { 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG "metric": { 09:47:24 DEBUG { 09:47:24 DEBUG "instance": "10.142.15.199:9100" 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG }, 09:47:24 DEBUG "data": { 09:47:24 DEBUG "value": [ 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG 09:47:24 DEBUG "result": [ 09:47:24 DEBUG 09:47:24 DEBUG { 09:47:24 DEBUG 09:47:24 DEBUG "metric": { 09:47:24 DEBUG [...] 09:47:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:47:24 DEBUG 09:47:24 DEBUG }, 09:47:24 DEBUG 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "instance": "10.142.15.212:9100" 09:47:24 DEBUG 09:47:24 DEBUG }, 09:47:24 DEBUG 09:47:24 DEBUG "value": [ 09:47:24 DEBUG [...] 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG 09:47:24 DEBUG "28634.199999999997" 09:47:24 DEBUG 09:47:24 DEBUG ] 09:47:24 DEBUG 09:47:24 DEBUG } 09:47:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:47:24 DEBUG ] 09:47:24 DEBUG }, 09:47:24 DEBUG } 09:47:24 DEBUG "value": [ 09:47:24 DEBUG } 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-idrepo-2" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0.014980484657504433" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263644" 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "instance": "10.142.15.199:9100" 09:47:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:47:24 DEBUG }, 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-idrepo-2" 09:47:24 DEBUG "instance": "10.142.15.212:9100" 09:47:24 DEBUG }, 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "4101.066022733966" 09:47:24 DEBUG "31425.333333333332" 09:47:24 DEBUG ] 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0.2" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "instance": "10.142.15.199:9100" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "instance": "10.142.15.212:9100" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263644" 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "instance": "10.142.15.199:9100" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "instance": "10.142.15.212:9100" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0.0024999999999998582" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "3.449397803128028" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263644" 09:47:24 INFO [http_cmd]: http status code OK 09:47:24 DEBUG --- status code --- 09:47:24 DEBUG http status code is 200 (expected 200) 09:47:24 DEBUG --- http response --- 09:47:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:47:24 DEBUG ----- output ----- 09:47:24 DEBUG { 09:47:24 DEBUG "status": "success", 09:47:24 DEBUG "data": { 09:47:24 DEBUG "resultType": "vector", 09:47:24 DEBUG "result": [ 09:47:24 DEBUG { 09:47:24 DEBUG "metric": { 09:47:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG [...] 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG 09:47:24 DEBUG "pod": "ds-idrepo-2" 09:47:24 DEBUG }, 09:47:24 DEBUG "value": [ 09:47:24 DEBUG 1685263644, 09:47:24 DEBUG "0" 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG ] 09:47:24 DEBUG } 09:47:24 DEBUG } 09:47:31 INFO 09:47:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:47:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:47:31 INFO [loop_until]: OK (rc = 0) 09:47:31 DEBUG --- stdout --- 09:47:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 6m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 8m 455Mi ds-cts-1 9m 409Mi ds-cts-2 8m 424Mi ds-idrepo-0 7218m 9897Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10661m 4528Mi idm-58dc667486-h86fb 9130m 4175Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3489m 2686Mi 09:47:31 DEBUG --- stderr --- 09:47:31 DEBUG 09:47:33 INFO 09:47:33 INFO [loop_until]: kubectl --namespace=xlou top node 09:47:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:47:33 INFO [loop_until]: OK (rc = 0) 09:47:33 DEBUG --- stdout --- 09:47:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 88m 0% 6852Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 67m 0% 6651Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 128m 0% 1536Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 77m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15431m 97% 5703Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9363m 58% 5302Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 80m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1089Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1097Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1073Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13169Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7343m 46% 10438Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14332Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3592m 22% 3638Mi 6% 09:47:33 DEBUG --- stderr --- 09:47:33 DEBUG 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "ds-idrepo-2" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0.6789373893991741" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "ds-idrepo-0" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "ds-idrepo-1" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263704" 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "ds-idrepo-0" 09:48:24 DEBUG "pod": "ds-idrepo-0" 09:48:24 DEBUG }, 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "ds-idrepo-1" 09:48:24 DEBUG "pod": "ds-cts-1" 09:48:24 DEBUG }, 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG "0" 09:48:24 DEBUG ] 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "ds-idrepo-0" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG "pod": "ds-cts-1" 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG }, 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG "value": [ 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG "0" 09:48:24 DEBUG { 09:48:24 DEBUG ] 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG } 09:48:24 DEBUG "data": { 09:48:24 DEBUG ] 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG } 09:48:24 DEBUG "result": [ 09:48:24 DEBUG } 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "4758032384" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "ds-idrepo-0" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "ds-idrepo-1" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:48:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:48:24 DEBUG }, 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "ds-idrepo-2" 09:48:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:48:24 DEBUG }, 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "2275.304356619196" 09:48:24 DEBUG "0.2" 09:48:24 DEBUG ] 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "ds-cts-2" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "ds-cts-1" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263704" 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:48:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:48:24 DEBUG }, 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG { 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "data": { 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG "result": [ 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG { 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "metric": { 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "instance": "10.142.15.199:9100" 09:48:24 DEBUG 09:48:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:48:24 DEBUG }, 09:48:24 DEBUG "pod": "ds-idrepo-2" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG 09:48:24 DEBUG "0.008917625751005647" 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263704" 09:48:24 DEBUG ] 09:48:24 DEBUG [...] 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG "instance": "10.142.15.212:9100" 09:48:24 DEBUG } 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "28478.5" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "instance": "10.142.15.199:9100" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "instance": "10.142.15.212:9100" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "31359.833333333332" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "instance": "10.142.15.199:9100" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "instance": "10.142.15.212:9100" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263704" 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:48:24 DEBUG "instance": "10.142.15.199:9100" 09:48:24 DEBUG }, 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:48:24 DEBUG "instance": "10.142.15.212:9100" 09:48:24 DEBUG }, 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG "0.0026000000000000988" 09:48:24 DEBUG ] 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "3.457574811971918" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263704" 09:48:24 INFO [http_cmd]: http status code OK 09:48:24 DEBUG --- status code --- 09:48:24 DEBUG http status code is 200 (expected 200) 09:48:24 DEBUG --- http response --- 09:48:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:48:24 DEBUG ----- output ----- 09:48:24 DEBUG { 09:48:24 DEBUG "status": "success", 09:48:24 DEBUG "data": { 09:48:24 DEBUG "resultType": "vector", 09:48:24 DEBUG "result": [ 09:48:24 DEBUG { 09:48:24 DEBUG "metric": { 09:48:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG [...] 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG 09:48:24 DEBUG "pod": "ds-idrepo-2" 09:48:24 DEBUG }, 09:48:24 DEBUG "value": [ 09:48:24 DEBUG 1685263704, 09:48:24 DEBUG "0" 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG ] 09:48:24 DEBUG } 09:48:24 DEBUG } 09:48:31 INFO 09:48:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:48:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:48:31 INFO [loop_until]: OK (rc = 0) 09:48:31 DEBUG --- stdout --- 09:48:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 8m 455Mi ds-cts-1 9m 409Mi ds-cts-2 8m 424Mi ds-idrepo-0 7017m 9897Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10131m 4533Mi idm-58dc667486-h86fb 9190m 4182Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3434m 2687Mi 09:48:31 DEBUG --- stderr --- 09:48:31 DEBUG 09:48:33 INFO 09:48:33 INFO [loop_until]: kubectl --namespace=xlou top node 09:48:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:48:34 INFO [loop_until]: OK (rc = 0) 09:48:34 DEBUG --- stdout --- 09:48:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 88m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 68m 0% 6656Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 128m 0% 1545Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 75m 0% 6822Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14428m 90% 5711Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9614m 60% 5310Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1352Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1088Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1098Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1074Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 65m 0% 13168Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7184m 45% 10440Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 68m 0% 14333Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3543m 22% 3624Mi 6% 09:48:34 DEBUG --- stderr --- 09:48:34 DEBUG 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "ds-idrepo-0" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "ds-cts-1" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "ds-idrepo-0" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG "pod": "ds-cts-1" 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG }, 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:49:24 DEBUG "value": [ 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG { 09:49:24 DEBUG "0" 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG ] 09:49:24 DEBUG "data": { 09:49:24 DEBUG } 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG ] 09:49:24 DEBUG "result": [ 09:49:24 DEBUG } 09:49:24 DEBUG { 09:49:24 DEBUG } 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "ds-idrepo-0" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "ds-idrepo-1" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "ds-idrepo-0" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "ds-idrepo-1" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "4763475968" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "ds-idrepo-0" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "ds-idrepo-1" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "ds-cts-2" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "ds-cts-1" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263764" 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "metric": { 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:49:24 DEBUG "instance": "10.142.15.199:9100" 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG }, 09:49:24 DEBUG }, 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG "value": [ 09:49:24 DEBUG "value": [ 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG { 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG [...] 09:49:24 DEBUG [...] 09:49:24 DEBUG "data": { 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "result": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG { 09:49:24 DEBUG "instance": "10.142.15.212:9100" 09:49:24 DEBUG "pod": "ds-idrepo-2" 09:49:24 DEBUG "metric": { 09:49:24 DEBUG }, 09:49:24 DEBUG }, 09:49:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:49:24 DEBUG "value": [ 09:49:24 DEBUG "value": [ 09:49:24 DEBUG }, 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG "28852.833333333332" 09:49:24 DEBUG "0" 09:49:24 DEBUG 09:49:24 DEBUG ] 09:49:24 DEBUG ] 09:49:24 DEBUG 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 DEBUG 09:49:24 DEBUG ] 09:49:24 DEBUG ] 09:49:24 DEBUG [...] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 DEBUG 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0.2" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "ds-idrepo-2" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "instance": "10.142.15.199:9100" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "instance": "10.142.15.212:9100" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "31490.866666666665" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "instance": "10.142.15.199:9100" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "instance": "10.142.15.212:9100" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0.0003333333333330302" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "instance": "10.142.15.199:9100" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "instance": "10.142.15.212:9100" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0.002800000000000343" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "3.3825764659536124" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263764" 09:49:24 INFO [http_cmd]: http status code OK 09:49:24 DEBUG --- status code --- 09:49:24 DEBUG http status code is 200 (expected 200) 09:49:24 DEBUG --- http response --- 09:49:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 09:49:24 DEBUG ----- output ----- 09:49:24 DEBUG { 09:49:24 DEBUG "status": "success", 09:49:24 DEBUG "data": { 09:49:24 DEBUG "resultType": "vector", 09:49:24 DEBUG "result": [ 09:49:24 DEBUG { 09:49:24 DEBUG "metric": { 09:49:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG [...] 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG 09:49:24 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:49:24 DEBUG }, 09:49:24 DEBUG "value": [ 09:49:24 DEBUG 1685263764, 09:49:24 DEBUG "0" 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG ] 09:49:24 DEBUG } 09:49:24 DEBUG } 09:49:31 INFO 09:49:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:49:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:49:31 INFO [loop_until]: OK (rc = 0) 09:49:31 DEBUG --- stdout --- 09:49:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5768Mi ds-cts-0 9m 456Mi ds-cts-1 8m 409Mi ds-cts-2 9m 424Mi ds-idrepo-0 7091m 9897Mi ds-idrepo-1 13m 13725Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10098m 4541Mi idm-58dc667486-h86fb 8957m 4189Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3292m 2688Mi 09:49:31 DEBUG --- stderr --- 09:49:31 DEBUG 09:49:34 INFO 09:49:34 INFO [loop_until]: kubectl --namespace=xlou top node 09:49:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:49:34 INFO [loop_until]: OK (rc = 0) 09:49:34 DEBUG --- stdout --- 09:49:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 89m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 69m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 128m 0% 1535Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 73m 0% 6823Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14140m 88% 5717Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9219m 58% 5315Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1352Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1093Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1100Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1076Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13166Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7171m 45% 10440Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14333Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3548m 22% 3625Mi 6% 09:49:34 DEBUG --- stderr --- 09:49:34 DEBUG 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "ds-idrepo-0" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "ds-cts-1" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "ds-idrepo-0" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "ds-cts-1" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "ds-cts-2" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "ds-idrepo-0" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "ds-idrepo-1" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG "data": { 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG "result": [ 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG { 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG "metric": { 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG "pod": "ds-idrepo-0" 09:50:24 DEBUG { 09:50:24 DEBUG }, 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "value": [ 09:50:24 DEBUG "data": { 09:50:24 DEBUG 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG 09:50:24 DEBUG "result": [ 09:50:24 DEBUG 09:50:24 DEBUG { 09:50:24 DEBUG [...] 09:50:24 DEBUG "metric": { 09:50:24 DEBUG 09:50:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:50:24 DEBUG 09:50:24 DEBUG }, 09:50:24 DEBUG 09:50:24 DEBUG "value": [ 09:50:24 DEBUG "pod": "ds-idrepo-1" 09:50:24 DEBUG 09:50:24 DEBUG }, 09:50:24 DEBUG 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG [...] 09:50:24 DEBUG "0" 09:50:24 DEBUG 09:50:24 DEBUG ] 09:50:24 DEBUG 09:50:24 DEBUG } 09:50:24 DEBUG 09:50:24 DEBUG ] 09:50:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:50:24 DEBUG } 09:50:24 DEBUG }, 09:50:24 DEBUG } 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "4773650432" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "ds-idrepo-0" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "ds-idrepo-1" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "ds-cts-2" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "ds-cts-1" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263824" 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG { 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG "data": { 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG { 09:50:24 DEBUG "result": [ 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG { 09:50:24 DEBUG "data": { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:50:24 DEBUG "result": [ 09:50:24 DEBUG }, 09:50:24 DEBUG { 09:50:24 DEBUG "value": [ 09:50:24 DEBUG "metric": { 09:50:24 DEBUG 09:50:24 DEBUG "instance": "10.142.15.199:9100" 09:50:24 DEBUG 09:50:24 DEBUG }, 09:50:24 DEBUG 09:50:24 DEBUG "value": [ 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:50:24 DEBUG 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG }, 09:50:24 DEBUG 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG { 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "instance": "10.142.15.212:9100" 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "0.2" 09:50:24 DEBUG }, 09:50:24 DEBUG "data": { 09:50:24 DEBUG ] 09:50:24 DEBUG "value": [ 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG } 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "result": [ 09:50:24 DEBUG ] 09:50:24 DEBUG "28467.033333333333" 09:50:24 DEBUG { 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG "metric": { 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:50:24 DEBUG ] 09:50:24 DEBUG }, 09:50:24 DEBUG } 09:50:24 DEBUG "value": [ 09:50:24 DEBUG } 09:50:24 DEBUG 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "ds-cts-2" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "13541172.365738293" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:50:24 DEBUG }, 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "ds-cts-2" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0.005828029833236705" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263824" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "instance": "10.142.15.199:9100" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "instance": "10.142.15.212:9100" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "instance": "10.142.15.199:9100" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "instance": "10.142.15.212:9100" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "31684.966666666667" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "3.4776684351907394" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "instance": "10.142.15.199:9100" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "instance": "10.142.15.212:9100" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0.0026333333333331175" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263824" 09:50:24 INFO [http_cmd]: http status code OK 09:50:24 DEBUG --- status code --- 09:50:24 DEBUG http status code is 200 (expected 200) 09:50:24 DEBUG --- http response --- 09:50:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:50:24 DEBUG ----- output ----- 09:50:24 DEBUG { 09:50:24 DEBUG "status": "success", 09:50:24 DEBUG "data": { 09:50:24 DEBUG "resultType": "vector", 09:50:24 DEBUG "result": [ 09:50:24 DEBUG { 09:50:24 DEBUG "metric": { 09:50:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG [...] 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG 09:50:24 DEBUG "pod": "ds-cts-2" 09:50:24 DEBUG }, 09:50:24 DEBUG "value": [ 09:50:24 DEBUG 1685263824, 09:50:24 DEBUG "0" 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG ] 09:50:24 DEBUG } 09:50:24 DEBUG } 09:50:31 INFO 09:50:31 INFO [loop_until]: kubectl --namespace=xlou top pods 09:50:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:50:32 INFO [loop_until]: OK (rc = 0) 09:50:32 DEBUG --- stdout --- 09:50:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 5m 5816Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 8m 458Mi ds-cts-1 8m 409Mi ds-cts-2 9m 424Mi ds-idrepo-0 6824m 9897Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 13m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 9826m 4548Mi idm-58dc667486-h86fb 9113m 4197Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3350m 2689Mi 09:50:32 DEBUG --- stderr --- 09:50:32 DEBUG 09:50:34 INFO 09:50:34 INFO [loop_until]: kubectl --namespace=xlou top node 09:50:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:50:34 INFO [loop_until]: OK (rc = 0) 09:50:34 DEBUG --- stdout --- 09:50:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 90m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 69m 0% 6656Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 117m 0% 1542Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 74m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14155m 89% 5721Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9081m 57% 5321Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1090Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1101Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1073Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13167Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6947m 43% 10438Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 68m 0% 14329Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3447m 21% 3626Mi 6% 09:50:34 DEBUG --- stderr --- 09:50:34 DEBUG 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "pod": "ds-idrepo-0" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "pod": "ds-cts-1" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "0.03333333333333333" 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG "data": { 09:51:24 DEBUG { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG "data": { 09:51:24 DEBUG { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "result": [ 09:51:24 DEBUG "pod": "ds-idrepo-0" 09:51:24 DEBUG { 09:51:24 DEBUG }, 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:51:24 DEBUG 09:51:24 DEBUG }, 09:51:24 DEBUG 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "pod": "ds-cts-1" 09:51:24 DEBUG 09:51:24 DEBUG }, 09:51:24 DEBUG 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG }, 09:51:24 DEBUG "0" 09:51:24 DEBUG "value": [ 09:51:24 DEBUG ] 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG } 09:51:24 DEBUG "0" 09:51:24 DEBUG ] 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "0" 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "pod": "ds-idrepo-0" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "pod": "ds-idrepo-1" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "0" 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263884" 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "pod": "ds-idrepo-0" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG "pod": "ds-idrepo-1" 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG }, 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG "value": [ 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:51:24 DEBUG "0" 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG ] 09:51:24 DEBUG { 09:51:24 DEBUG } 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG ] 09:51:24 DEBUG "data": { 09:51:24 DEBUG } 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG } 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "4782243840" 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "pod": "ds-idrepo-0" 09:51:24 DEBUG "pod": "ds-cts-2" 09:51:24 DEBUG }, 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "pod": "ds-idrepo-1" 09:51:24 DEBUG "pod": "ds-cts-1" 09:51:24 DEBUG }, 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "0" 09:51:24 DEBUG "0" 09:51:24 DEBUG ] 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263884" 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG { 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "data": { 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG "result": [ 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG { 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:51:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:51:24 DEBUG "instance": "10.142.15.199:9100" 09:51:24 DEBUG }, 09:51:24 DEBUG }, 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG [...] 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "pod": "ds-cts-0" 09:51:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:51:24 DEBUG "instance": "10.142.15.212:9100" 09:51:24 DEBUG }, 09:51:24 DEBUG }, 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "0" 09:51:24 DEBUG "0.2" 09:51:24 DEBUG "28380.86666666667" 09:51:24 DEBUG ] 09:51:24 DEBUG ] 09:51:24 DEBUG ] 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263884" 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG ] 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263884" 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:51:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:51:24 DEBUG }, 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:51:24 DEBUG "pod": "ds-cts-0" 09:51:24 DEBUG }, 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "0" 09:51:24 DEBUG "0" 09:51:24 DEBUG ] 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "0" 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263884" 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "instance": "10.142.15.199:9100" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "instance": "10.142.15.212:9100" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "0" 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "instance": "10.142.15.199:9100" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "instance": "10.142.15.212:9100" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "31318.13333333333" 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263884" 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "3.3787006404455995" 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263884" 09:51:24 INFO [http_cmd]: http status code OK 09:51:24 DEBUG --- status code --- 09:51:24 DEBUG http status code is 200 (expected 200) 09:51:24 DEBUG --- http response --- 09:51:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:51:24 DEBUG ----- output ----- 09:51:24 DEBUG { 09:51:24 DEBUG "status": "success", 09:51:24 DEBUG "data": { 09:51:24 DEBUG "resultType": "vector", 09:51:24 DEBUG "result": [ 09:51:24 DEBUG { 09:51:24 DEBUG "metric": { 09:51:24 DEBUG "instance": "10.142.15.199:9100" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG [...] 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG 09:51:24 DEBUG "instance": "10.142.15.212:9100" 09:51:24 DEBUG }, 09:51:24 DEBUG "value": [ 09:51:24 DEBUG 1685263884, 09:51:24 DEBUG "0.0025666666666663692" 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG ] 09:51:24 DEBUG } 09:51:24 DEBUG } 09:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263885" 09:51:25 INFO [http_cmd]: http status code OK 09:51:25 DEBUG --- status code --- 09:51:25 DEBUG http status code is 200 (expected 200) 09:51:25 DEBUG --- http response --- 09:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:51:25 DEBUG ----- output ----- 09:51:25 DEBUG { 09:51:25 DEBUG "status": "success", 09:51:25 DEBUG "data": { 09:51:25 DEBUG "resultType": "vector", 09:51:25 DEBUG "result": [ 09:51:25 DEBUG { 09:51:25 DEBUG "metric": { 09:51:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:51:25 DEBUG }, 09:51:25 DEBUG "value": [ 09:51:25 DEBUG 09:51:25 DEBUG 09:51:25 DEBUG 09:51:25 DEBUG [...] 09:51:25 DEBUG 09:51:25 DEBUG 09:51:25 DEBUG 09:51:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:51:25 DEBUG }, 09:51:25 DEBUG "value": [ 09:51:25 DEBUG 1685263885, 09:51:25 DEBUG "0" 09:51:25 DEBUG ] 09:51:25 DEBUG } 09:51:25 DEBUG ] 09:51:25 DEBUG } 09:51:25 DEBUG } 09:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263885" 09:51:25 INFO [http_cmd]: http status code OK 09:51:25 DEBUG --- status code --- 09:51:25 DEBUG http status code is 200 (expected 200) 09:51:25 DEBUG --- http response --- 09:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:51:25 DEBUG ----- output ----- 09:51:25 DEBUG { 09:51:25 DEBUG "status": "success", 09:51:25 DEBUG "data": { 09:51:25 DEBUG "resultType": "vector", 09:51:25 DEBUG "result": [ 09:51:25 DEBUG { 09:51:25 DEBUG "metric": { 09:51:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:51:25 DEBUG }, 09:51:25 DEBUG "value": [ 09:51:25 DEBUG 09:51:25 DEBUG 09:51:25 DEBUG 09:51:25 DEBUG [...] 09:51:25 DEBUG 09:51:25 DEBUG 09:51:25 DEBUG 09:51:25 DEBUG "pod": "ds-cts-2" 09:51:25 DEBUG }, 09:51:25 DEBUG "value": [ 09:51:25 DEBUG 1685263885, 09:51:25 DEBUG "0" 09:51:25 DEBUG ] 09:51:25 DEBUG } 09:51:25 DEBUG ] 09:51:25 DEBUG } 09:51:25 DEBUG } 09:51:32 INFO 09:51:32 INFO [loop_until]: kubectl --namespace=xlou top pods 09:51:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:51:32 INFO [loop_until]: OK (rc = 0) 09:51:32 DEBUG --- stdout --- 09:51:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5768Mi ds-cts-0 9m 458Mi ds-cts-1 8m 409Mi ds-cts-2 8m 425Mi ds-idrepo-0 7112m 9897Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10096m 4556Mi idm-58dc667486-h86fb 9439m 4203Mi lodemon-cfd7686d9-6dlt4 1m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3394m 2690Mi 09:51:32 DEBUG --- stderr --- 09:51:32 DEBUG 09:51:34 INFO 09:51:34 INFO [loop_until]: kubectl --namespace=xlou top node 09:51:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:51:34 INFO [loop_until]: OK (rc = 0) 09:51:34 DEBUG --- stdout --- 09:51:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 91m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 67m 0% 6655Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 123m 0% 1543Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 72m 0% 6816Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14545m 91% 5726Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9228m 58% 5341Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1353Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1085Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1099Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1070Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13167Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7158m 45% 10439Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 64m 0% 14334Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3524m 22% 3630Mi 6% 09:51:34 DEBUG --- stderr --- 09:51:34 DEBUG 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG { 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "data": { 09:52:24 DEBUG "resultType": "vector", 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 DEBUG "result": [ 09:52:24 DEBUG { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "pod": "ds-idrepo-0" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG [...] 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "pod": "ds-cts-1" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG "0" 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "data": { 09:52:24 DEBUG "data": { 09:52:24 DEBUG "data": { 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "result": [ 09:52:24 DEBUG "result": [ 09:52:24 DEBUG "result": [ 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "pod": "ds-idrepo-0" 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG "pod": "ds-idrepo-0" 09:52:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:52:24 DEBUG }, 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG }, 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG "value": [ 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG [...] 09:52:24 DEBUG { 09:52:24 DEBUG [...] 09:52:24 DEBUG [...] 09:52:24 DEBUG 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "data": { 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "pod": "ds-cts-1" 09:52:24 DEBUG "result": [ 09:52:24 DEBUG "pod": "ds-idrepo-1" 09:52:24 DEBUG "pod": "ds-cts-0" 09:52:24 DEBUG }, 09:52:24 DEBUG { 09:52:24 DEBUG }, 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "value": [ 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG "0" 09:52:24 DEBUG }, 09:52:24 DEBUG "0" 09:52:24 DEBUG "0" 09:52:24 DEBUG ] 09:52:24 DEBUG "value": [ 09:52:24 DEBUG ] 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG 09:52:24 DEBUG ] 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG [...] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG "0" 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG { 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "data": { 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "result": [ 09:52:24 DEBUG { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "pod": "ds-idrepo-0" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG [...] 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "pod": "ds-idrepo-1" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG "0" 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685263944" 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG { 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "data": { 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "result": [ 09:52:24 DEBUG { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG [...] 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG "4790472704" 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG { 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "data": { 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "result": [ 09:52:24 DEBUG { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "pod": "ds-cts-2" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG [...] 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "pod": "ds-cts-1" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG "0" 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG ] 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG } 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:24 DEBUG ] 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG } 09:52:24 DEBUG { 09:52:24 DEBUG } 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "data": { 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "result": [ 09:52:24 DEBUG { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "pod": "ds-idrepo-0" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG [...] 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "pod": "ds-idrepo-1" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG "0" 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG { 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "data": { 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "result": [ 09:52:24 DEBUG { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG [...] 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG "0.2" 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263944" 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263944" 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263944" 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG ----- output ----- 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG { 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG { 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG "data": { 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG "data": { 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG "result": [ 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG "result": [ 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "data": { 09:52:24 DEBUG "data": { 09:52:24 DEBUG "data": { 09:52:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:52:24 DEBUG "data": { 09:52:24 DEBUG "instance": "10.142.15.199:9100" 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG }, 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG }, 09:52:24 DEBUG "result": [ 09:52:24 DEBUG "result": [ 09:52:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263944" 09:52:24 DEBUG "result": [ 09:52:24 DEBUG "value": [ 09:52:24 DEBUG "result": [ 09:52:24 DEBUG "value": [ 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG { 09:52:24 DEBUG 09:52:24 DEBUG { 09:52:24 DEBUG 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG 09:52:24 DEBUG "metric": { 09:52:24 DEBUG 09:52:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:52:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:52:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:52:24 DEBUG 09:52:24 DEBUG "instance": "10.142.15.199:9100" 09:52:24 DEBUG 09:52:24 DEBUG }, 09:52:24 DEBUG }, 09:52:24 DEBUG }, 09:52:24 DEBUG [...] 09:52:24 DEBUG }, 09:52:24 DEBUG [...] 09:52:24 DEBUG "value": [ 09:52:24 DEBUG "value": [ 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "pod": "ds-cts-0" 09:52:24 DEBUG 09:52:24 DEBUG "instance": "10.142.15.212:9100" 09:52:24 DEBUG [...] 09:52:24 DEBUG [...] 09:52:24 DEBUG [...] 09:52:24 DEBUG }, 09:52:24 DEBUG [...] 09:52:24 DEBUG }, 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "0.005369553953332205" 09:52:24 DEBUG 09:52:24 DEBUG "28595.93333333333" 09:52:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:52:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:52:24 DEBUG "pod": "ds-cts-0" 09:52:24 DEBUG ] 09:52:24 DEBUG "instance": "10.142.15.212:9100" 09:52:24 DEBUG ] 09:52:24 DEBUG }, 09:52:24 DEBUG }, 09:52:24 DEBUG }, 09:52:24 DEBUG } 09:52:24 DEBUG }, 09:52:24 DEBUG } 09:52:24 DEBUG "value": [ 09:52:24 DEBUG "value": [ 09:52:24 DEBUG "value": [ 09:52:24 DEBUG ] 09:52:24 DEBUG "value": [ 09:52:24 DEBUG ] 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG } 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG } 09:52:24 DEBUG "0" 09:52:24 DEBUG "0" 09:52:24 DEBUG "12504459.518218897" 09:52:24 DEBUG } 09:52:24 DEBUG "0" 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG ] 09:52:24 DEBUG ] 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 INFO [http_cmd]: http status code OK 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG ] 09:52:24 DEBUG --- status code --- 09:52:24 DEBUG ] 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG http status code is 200 (expected 200) 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG --- http response --- 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:52:24 DEBUG ----- output ----- 09:52:24 DEBUG { 09:52:24 DEBUG "status": "success", 09:52:24 DEBUG "data": { 09:52:24 DEBUG "resultType": "vector", 09:52:24 DEBUG "result": [ 09:52:24 DEBUG { 09:52:24 DEBUG "metric": { 09:52:24 DEBUG "instance": "10.142.15.199:9100" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG [...] 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG 09:52:24 DEBUG "instance": "10.142.15.212:9100" 09:52:24 DEBUG }, 09:52:24 DEBUG "value": [ 09:52:24 DEBUG 1685263944, 09:52:24 DEBUG "31345.699999999997" 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG ] 09:52:24 DEBUG } 09:52:24 DEBUG } 09:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685263945" 09:52:25 INFO [http_cmd]: http status code OK 09:52:25 DEBUG --- status code --- 09:52:25 DEBUG http status code is 200 (expected 200) 09:52:25 DEBUG --- http response --- 09:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:52:25 DEBUG ----- output ----- 09:52:25 DEBUG { 09:52:25 DEBUG "status": "success", 09:52:25 DEBUG "data": { 09:52:25 DEBUG "resultType": "vector", 09:52:25 DEBUG "result": [ 09:52:25 DEBUG { 09:52:25 DEBUG "metric": { 09:52:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:52:25 DEBUG }, 09:52:25 DEBUG "value": [ 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG [...] 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:52:25 DEBUG }, 09:52:25 DEBUG "value": [ 09:52:25 DEBUG 1685263945, 09:52:25 DEBUG "3.4957394080314055" 09:52:25 DEBUG ] 09:52:25 DEBUG } 09:52:25 DEBUG ] 09:52:25 DEBUG } 09:52:25 DEBUG } 09:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685263945" 09:52:25 INFO [http_cmd]: http status code OK 09:52:25 DEBUG --- status code --- 09:52:25 DEBUG http status code is 200 (expected 200) 09:52:25 DEBUG --- http response --- 09:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:52:25 DEBUG ----- output ----- 09:52:25 DEBUG { 09:52:25 DEBUG "status": "success", 09:52:25 DEBUG "data": { 09:52:25 DEBUG "resultType": "vector", 09:52:25 DEBUG "result": [ 09:52:25 DEBUG { 09:52:25 DEBUG "metric": { 09:52:25 DEBUG "instance": "10.142.15.199:9100" 09:52:25 DEBUG }, 09:52:25 DEBUG "value": [ 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG [...] 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG "instance": "10.142.15.212:9100" 09:52:25 DEBUG }, 09:52:25 DEBUG "value": [ 09:52:25 DEBUG 1685263945, 09:52:25 DEBUG "0.0026000000000000983" 09:52:25 DEBUG ] 09:52:25 DEBUG } 09:52:25 DEBUG ] 09:52:25 DEBUG } 09:52:25 DEBUG } 09:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263945" 09:52:25 INFO [http_cmd]: http status code OK 09:52:25 DEBUG --- status code --- 09:52:25 DEBUG http status code is 200 (expected 200) 09:52:25 DEBUG --- http response --- 09:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:52:25 DEBUG ----- output ----- 09:52:25 DEBUG { 09:52:25 DEBUG "status": "success", 09:52:25 DEBUG "data": { 09:52:25 DEBUG "resultType": "vector", 09:52:25 DEBUG "result": [ 09:52:25 DEBUG { 09:52:25 DEBUG "metric": { 09:52:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:52:25 DEBUG }, 09:52:25 DEBUG "value": [ 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG [...] 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG "pod": "ds-cts-0" 09:52:25 DEBUG }, 09:52:25 DEBUG "value": [ 09:52:25 DEBUG 1685263945, 09:52:25 DEBUG "0" 09:52:25 DEBUG ] 09:52:25 DEBUG } 09:52:25 DEBUG ] 09:52:25 DEBUG } 09:52:25 DEBUG } 09:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685263945" 09:52:25 INFO [http_cmd]: http status code OK 09:52:25 DEBUG --- status code --- 09:52:25 DEBUG http status code is 200 (expected 200) 09:52:25 DEBUG --- http response --- 09:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:25 DEBUG ----- output ----- 09:52:25 DEBUG { 09:52:25 DEBUG "status": "success", 09:52:25 DEBUG "data": { 09:52:25 DEBUG "resultType": "vector", 09:52:25 DEBUG "result": [ 09:52:25 DEBUG { 09:52:25 DEBUG "metric": { 09:52:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:52:25 DEBUG }, 09:52:25 DEBUG "value": [ 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG [...] 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG 09:52:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:52:25 DEBUG }, 09:52:25 DEBUG "value": [ 09:52:25 DEBUG 1685263945, 09:52:25 DEBUG "0" 09:52:25 DEBUG ] 09:52:25 DEBUG } 09:52:25 DEBUG ] 09:52:25 DEBUG } 09:52:25 DEBUG } 09:52:32 INFO 09:52:32 INFO [loop_until]: kubectl --namespace=xlou top pods 09:52:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:52:32 INFO [loop_until]: OK (rc = 0) 09:52:32 DEBUG --- stdout --- 09:52:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 8m 458Mi ds-cts-1 9m 409Mi ds-cts-2 8m 424Mi ds-idrepo-0 6753m 9897Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 13m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 9992m 4564Mi idm-58dc667486-h86fb 8846m 4212Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3388m 2691Mi 09:52:32 DEBUG --- stderr --- 09:52:32 DEBUG 09:52:34 INFO 09:52:34 INFO [loop_until]: kubectl --namespace=xlou top node 09:52:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:52:34 INFO [loop_until]: OK (rc = 0) 09:52:34 DEBUG --- stdout --- 09:52:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 88m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 67m 0% 6658Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 136m 0% 1548Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 72m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 13916m 87% 5732Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9213m 57% 5333Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 81m 0% 1352Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 63m 0% 1101Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 55m 0% 1100Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1071Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13167Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6933m 43% 10442Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14331Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3500m 22% 3632Mi 6% 09:52:34 DEBUG --- stderr --- 09:52:34 DEBUG 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG { 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG "data": { 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG "result": [ 09:53:24 DEBUG { 09:53:24 DEBUG "metric": { 09:53:24 DEBUG "pod": "ds-idrepo-0" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG [...] 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG "pod": "ds-cts-1" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG "0" 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG } 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG { 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG "data": { 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG "result": [ 09:53:24 DEBUG { 09:53:24 DEBUG "metric": { 09:53:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG [...] 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG "0" 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG } 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG { 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG "data": { 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG "result": [ 09:53:24 DEBUG { 09:53:24 DEBUG { 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG "metric": { 09:53:24 DEBUG "data": { 09:53:24 DEBUG "pod": "ds-idrepo-0" 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG }, 09:53:24 DEBUG "result": [ 09:53:24 DEBUG "value": [ 09:53:24 DEBUG { 09:53:24 DEBUG 09:53:24 DEBUG "metric": { 09:53:24 DEBUG 09:53:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:53:24 DEBUG 09:53:24 DEBUG }, 09:53:24 DEBUG [...] 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG "pod": "ds-idrepo-1" 09:53:24 DEBUG [...] 09:53:24 DEBUG }, 09:53:24 DEBUG 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG 09:53:24 DEBUG "0" 09:53:24 DEBUG "pod": "ds-cts-0" 09:53:24 DEBUG ] 09:53:24 DEBUG }, 09:53:24 DEBUG } 09:53:24 DEBUG "value": [ 09:53:24 DEBUG ] 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG } 09:53:24 DEBUG "0" 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG } 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG { 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG "data": { 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG "result": [ 09:53:24 DEBUG { 09:53:24 DEBUG "metric": { 09:53:24 DEBUG "pod": "ds-idrepo-0" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG [...] 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG "pod": "ds-cts-1" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG "0" 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG } 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG { 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG "data": { 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG "result": [ 09:53:24 DEBUG { 09:53:24 DEBUG "metric": { 09:53:24 DEBUG "pod": "ds-idrepo-0" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG [...] 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG "pod": "ds-idrepo-1" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG "0" 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG } 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264004" 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG { 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG "data": { 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG "result": [ 09:53:24 DEBUG { 09:53:24 DEBUG "metric": { 09:53:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG [...] 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG "4798124032" 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG } 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG { 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG "data": { 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG "result": [ 09:53:24 DEBUG { 09:53:24 DEBUG "metric": { 09:53:24 DEBUG "pod": "ds-cts-2" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG [...] 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG "pod": "ds-cts-1" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG "0" 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG } 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG { 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG "data": { 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG "result": [ 09:53:24 DEBUG { 09:53:24 DEBUG "metric": { 09:53:24 DEBUG "pod": "ds-idrepo-0" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG [...] 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG "pod": "ds-idrepo-1" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG "0" 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG } 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:24 INFO [http_cmd]: http status code OK 09:53:24 DEBUG --- status code --- 09:53:24 DEBUG http status code is 200 (expected 200) 09:53:24 DEBUG --- http response --- 09:53:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:53:24 DEBUG ----- output ----- 09:53:24 DEBUG { 09:53:24 DEBUG "status": "success", 09:53:24 DEBUG "data": { 09:53:24 DEBUG "resultType": "vector", 09:53:24 DEBUG "result": [ 09:53:24 DEBUG { 09:53:24 DEBUG "metric": { 09:53:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG [...] 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG 09:53:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:53:24 DEBUG }, 09:53:24 DEBUG "value": [ 09:53:24 DEBUG 1685264004, 09:53:24 DEBUG "0.2" 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG ] 09:53:24 DEBUG } 09:53:24 DEBUG } 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264004" 09:53:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264004" 09:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264005" 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG { 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "data": { 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "result": [ 09:53:25 DEBUG { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "instance": "10.142.15.199:9100" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG [...] 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG "instance": "10.142.15.212:9100" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 1685264004, 09:53:25 DEBUG "0.0003333333333330302" 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG { 09:53:25 DEBUG { 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "data": { 09:53:25 DEBUG "data": { 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "result": [ 09:53:25 DEBUG "result": [ 09:53:25 DEBUG { 09:53:25 DEBUG { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:53:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:53:25 DEBUG }, 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG [...] 09:53:25 DEBUG [...] 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG "pod": "ds-cts-0" 09:53:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:53:25 DEBUG }, 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 1685264004, 09:53:25 DEBUG 1685264005, 09:53:25 DEBUG "1257.5132184888282" 09:53:25 DEBUG "0.006431014131221694" 09:53:25 DEBUG ] 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 DEBUG ] 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264005" 09:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264005" 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG { 09:53:25 DEBUG { 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "data": { 09:53:25 DEBUG "data": { 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "result": [ 09:53:25 DEBUG "result": [ 09:53:25 DEBUG { 09:53:25 DEBUG { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "instance": "10.142.15.199:9100" 09:53:25 DEBUG "instance": "10.142.15.199:9100" 09:53:25 DEBUG }, 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG [...] 09:53:25 DEBUG [...] 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG "instance": "10.142.15.212:9100" 09:53:25 DEBUG "instance": "10.142.15.212:9100" 09:53:25 DEBUG }, 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 1685264005, 09:53:25 DEBUG 1685264005, 09:53:25 DEBUG "0.0024333333333331096" 09:53:25 DEBUG "28345.866666666665" 09:53:25 DEBUG ] 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 DEBUG ] 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264005" 09:53:25 DEBUG } 09:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264005" 09:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264005" 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG { 09:53:25 DEBUG { 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "data": { 09:53:25 DEBUG "data": { 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "result": [ 09:53:25 DEBUG "result": [ 09:53:25 DEBUG { 09:53:25 DEBUG { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:53:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:53:25 DEBUG }, 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG [...] 09:53:25 DEBUG [...] 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG 09:53:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:53:25 DEBUG }, 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 1685264005, 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG 1685264005, 09:53:25 DEBUG "0" 09:53:25 DEBUG { 09:53:25 DEBUG "0" 09:53:25 DEBUG ] 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG "data": { 09:53:25 DEBUG } 09:53:25 DEBUG ] 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG "result": [ 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 DEBUG { 09:53:25 DEBUG } 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "instance": "10.142.15.199:9100" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG [...] 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG "instance": "10.142.15.212:9100" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 1685264005, 09:53:25 DEBUG "28138.36666666667" 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264005" 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG { 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "data": { 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "result": [ 09:53:25 DEBUG { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG [...] 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 1685264005, 09:53:25 DEBUG "3.455526857789465" 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264005" 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG { 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "data": { 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "result": [ 09:53:25 DEBUG { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG [...] 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 1685264005, 09:53:25 DEBUG "0" 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264005" 09:53:25 INFO [http_cmd]: http status code OK 09:53:25 DEBUG --- status code --- 09:53:25 DEBUG http status code is 200 (expected 200) 09:53:25 DEBUG --- http response --- 09:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:53:25 DEBUG ----- output ----- 09:53:25 DEBUG { 09:53:25 DEBUG "status": "success", 09:53:25 DEBUG "data": { 09:53:25 DEBUG "resultType": "vector", 09:53:25 DEBUG "result": [ 09:53:25 DEBUG { 09:53:25 DEBUG "metric": { 09:53:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG [...] 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG 09:53:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:53:25 DEBUG }, 09:53:25 DEBUG "value": [ 09:53:25 DEBUG 1685264005, 09:53:25 DEBUG "0" 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG ] 09:53:25 DEBUG } 09:53:25 DEBUG } 09:53:32 INFO 09:53:32 INFO [loop_until]: kubectl --namespace=xlou top pods 09:53:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:53:32 INFO [loop_until]: OK (rc = 0) 09:53:32 DEBUG --- stdout --- 09:53:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 8m 458Mi ds-cts-1 8m 409Mi ds-cts-2 8m 424Mi ds-idrepo-0 6831m 9898Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 13m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 9760m 4573Mi idm-58dc667486-h86fb 9177m 4221Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3414m 2692Mi 09:53:32 DEBUG --- stderr --- 09:53:32 DEBUG 09:53:34 INFO 09:53:34 INFO [loop_until]: kubectl --namespace=xlou top node 09:53:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:53:34 INFO [loop_until]: OK (rc = 0) 09:53:34 DEBUG --- stdout --- 09:53:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 87m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 69m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 121m 0% 1545Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 73m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14422m 90% 5738Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9670m 60% 5348Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 80m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1090Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1101Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1074Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 61m 0% 13169Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7065m 44% 10442Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14332Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3557m 22% 3630Mi 6% 09:53:34 DEBUG --- stderr --- 09:53:34 DEBUG 09:54:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264064" 09:54:24 INFO [http_cmd]: http status code OK 09:54:24 DEBUG --- status code --- 09:54:24 DEBUG http status code is 200 (expected 200) 09:54:24 DEBUG --- http response --- 09:54:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:54:24 DEBUG ----- output ----- 09:54:24 DEBUG { 09:54:24 DEBUG "status": "success", 09:54:24 DEBUG "data": { 09:54:24 DEBUG "resultType": "vector", 09:54:24 DEBUG "result": [ 09:54:24 DEBUG { 09:54:24 DEBUG "metric": { 09:54:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:54:24 DEBUG }, 09:54:24 DEBUG "value": [ 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG [...] 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:54:24 DEBUG }, 09:54:24 DEBUG "value": [ 09:54:24 DEBUG 1685264064, 09:54:24 DEBUG "0" 09:54:24 DEBUG ] 09:54:24 DEBUG } 09:54:24 DEBUG ] 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264064" 09:54:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264064" 09:54:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264064" 09:54:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264064" 09:54:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264064" 09:54:24 INFO [http_cmd]: http status code OK 09:54:24 DEBUG --- status code --- 09:54:24 DEBUG http status code is 200 (expected 200) 09:54:24 DEBUG --- http response --- 09:54:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:54:24 DEBUG ----- output ----- 09:54:24 DEBUG { 09:54:24 DEBUG "status": "success", 09:54:24 DEBUG "data": { 09:54:24 DEBUG "resultType": "vector", 09:54:24 DEBUG "result": [ 09:54:24 DEBUG { 09:54:24 DEBUG "metric": { 09:54:24 DEBUG "pod": "ds-idrepo-0" 09:54:24 DEBUG }, 09:54:24 DEBUG "value": [ 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG [...] 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG "pod": "ds-cts-1" 09:54:24 DEBUG }, 09:54:24 DEBUG "value": [ 09:54:24 DEBUG 1685264064, 09:54:24 DEBUG "0" 09:54:24 DEBUG ] 09:54:24 DEBUG } 09:54:24 DEBUG ] 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 INFO [http_cmd]: http status code OK 09:54:24 DEBUG --- status code --- 09:54:24 INFO [http_cmd]: http status code OK 09:54:24 INFO [http_cmd]: http status code OK 09:54:24 INFO [http_cmd]: http status code OK 09:54:24 DEBUG http status code is 200 (expected 200) 09:54:24 DEBUG --- status code --- 09:54:24 DEBUG --- status code --- 09:54:24 DEBUG --- status code --- 09:54:24 DEBUG --- http response --- 09:54:24 DEBUG http status code is 200 (expected 200) 09:54:24 DEBUG http status code is 200 (expected 200) 09:54:24 DEBUG http status code is 200 (expected 200) 09:54:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:54:24 DEBUG --- http response --- 09:54:24 DEBUG --- http response --- 09:54:24 DEBUG --- http response --- 09:54:24 DEBUG ----- output ----- 09:54:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:54:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:54:24 DEBUG { 09:54:24 DEBUG ----- output ----- 09:54:24 DEBUG ----- output ----- 09:54:24 DEBUG ----- output ----- 09:54:24 DEBUG "status": "success", 09:54:24 DEBUG { 09:54:24 DEBUG { 09:54:24 DEBUG { 09:54:24 DEBUG "data": { 09:54:24 DEBUG "status": "success", 09:54:24 DEBUG "status": "success", 09:54:24 DEBUG "status": "success", 09:54:24 DEBUG "resultType": "vector", 09:54:24 DEBUG "data": { 09:54:24 DEBUG "data": { 09:54:24 DEBUG "data": { 09:54:24 DEBUG "result": [ 09:54:24 DEBUG "resultType": "vector", 09:54:24 DEBUG "resultType": "vector", 09:54:24 DEBUG "resultType": "vector", 09:54:24 DEBUG { 09:54:24 DEBUG "result": [ 09:54:24 DEBUG "result": [ 09:54:24 DEBUG "result": [ 09:54:24 DEBUG "metric": { 09:54:24 DEBUG { 09:54:24 DEBUG { 09:54:24 DEBUG { 09:54:24 DEBUG "pod": "ds-idrepo-0" 09:54:24 DEBUG "metric": { 09:54:24 DEBUG "metric": { 09:54:24 DEBUG "metric": { 09:54:24 DEBUG }, 09:54:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:54:24 DEBUG "pod": "ds-idrepo-0" 09:54:24 DEBUG "pod": "ds-idrepo-0" 09:54:24 DEBUG "value": [ 09:54:24 DEBUG }, 09:54:24 DEBUG }, 09:54:24 DEBUG }, 09:54:24 DEBUG 09:54:24 DEBUG "value": [ 09:54:24 DEBUG "value": [ 09:54:24 DEBUG "value": [ 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG [...] 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG [...] 09:54:24 DEBUG [...] 09:54:24 DEBUG [...] 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG "pod": "ds-idrepo-1" 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG }, 09:54:24 DEBUG "pod": "ds-idrepo-2" 09:54:24 DEBUG "pod": "ds-cts-1" 09:54:24 DEBUG "pod": "ds-idrepo-1" 09:54:24 DEBUG "value": [ 09:54:24 DEBUG }, 09:54:24 DEBUG }, 09:54:24 DEBUG }, 09:54:24 DEBUG 1685264064, 09:54:24 DEBUG "value": [ 09:54:24 DEBUG "value": [ 09:54:24 DEBUG "value": [ 09:54:24 DEBUG "0" 09:54:24 DEBUG 1685264064, 09:54:24 DEBUG 1685264064, 09:54:24 DEBUG 1685264064, 09:54:24 DEBUG ] 09:54:24 DEBUG "0.935031436401741" 09:54:24 DEBUG "0" 09:54:24 DEBUG "0" 09:54:24 DEBUG } 09:54:24 DEBUG ] 09:54:24 DEBUG ] 09:54:24 DEBUG ] 09:54:24 DEBUG ] 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 DEBUG ] 09:54:24 DEBUG ] 09:54:24 DEBUG ] 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264064" 09:54:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264064" 09:54:24 INFO [http_cmd]: http status code OK 09:54:24 DEBUG --- status code --- 09:54:24 DEBUG http status code is 200 (expected 200) 09:54:24 DEBUG --- http response --- 09:54:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:24 DEBUG ----- output ----- 09:54:24 DEBUG { 09:54:24 DEBUG "status": "success", 09:54:24 DEBUG "data": { 09:54:24 DEBUG "resultType": "vector", 09:54:24 DEBUG "result": [ 09:54:24 DEBUG { 09:54:24 DEBUG "metric": { 09:54:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:54:24 DEBUG }, 09:54:24 DEBUG "value": [ 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG [...] 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG 09:54:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:54:24 DEBUG }, 09:54:24 DEBUG "value": [ 09:54:24 DEBUG 1685264064, 09:54:24 DEBUG "4809383936" 09:54:24 DEBUG ] 09:54:24 DEBUG } 09:54:24 DEBUG ] 09:54:24 DEBUG } 09:54:24 DEBUG } 09:54:24 INFO [http_cmd]: http status code OK 09:54:24 DEBUG --- status code --- 09:54:24 DEBUG http status code is 200 (expected 200) 09:54:24 DEBUG --- http response --- 09:54:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:54:24 DEBUG ----- output ----- 09:54:24 DEBUG { 09:54:24 DEBUG "status": "success", 09:54:24 DEBUG "data": { 09:54:24 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "pod": "ds-cts-2" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "pod": "ds-cts-1" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264064, 09:54:25 DEBUG "0" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264065" 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264065" 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "pod": "ds-idrepo-0" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "pod": "ds-idrepo-1" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "0" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264065" 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "instance": "10.142.15.199:9100" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "instance": "10.142.15.212:9100" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "0" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "0.2" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264065" 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264065" 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264065" 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264065" 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "0" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG { 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "data": { 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG "result": [ 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG { 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "instance": "10.142.15.199:9100" 09:54:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:54:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:54:25 DEBUG }, 09:54:25 DEBUG }, 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG "value": [ 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG [...] 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "instance": "10.142.15.212:9100" 09:54:25 DEBUG "pod": "ds-idrepo-2" 09:54:25 DEBUG "pod": "ds-idrepo-2" 09:54:25 DEBUG }, 09:54:25 DEBUG }, 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG "value": [ 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "26298.86666666667" 09:54:25 DEBUG "0.012875521134944973" 09:54:25 DEBUG "5414.67032081251" 09:54:25 DEBUG ] 09:54:25 DEBUG ] 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG ] 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264065" 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264065" 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "0" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "instance": "10.142.15.199:9100" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "instance": "10.142.15.212:9100" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "0.002533333333333587" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264065" 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "instance": "10.142.15.199:9100" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "instance": "10.142.15.212:9100" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "28006.133333333335" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264065" 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "3.478160136172548" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264065" 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "pod": "ds-idrepo-2" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "0" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264065" 09:54:25 INFO [http_cmd]: http status code OK 09:54:25 DEBUG --- status code --- 09:54:25 DEBUG http status code is 200 (expected 200) 09:54:25 DEBUG --- http response --- 09:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:54:25 DEBUG ----- output ----- 09:54:25 DEBUG { 09:54:25 DEBUG "status": "success", 09:54:25 DEBUG "data": { 09:54:25 DEBUG "resultType": "vector", 09:54:25 DEBUG "result": [ 09:54:25 DEBUG { 09:54:25 DEBUG "metric": { 09:54:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG [...] 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG 09:54:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:54:25 DEBUG }, 09:54:25 DEBUG "value": [ 09:54:25 DEBUG 1685264065, 09:54:25 DEBUG "0" 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG ] 09:54:25 DEBUG } 09:54:25 DEBUG } 09:54:32 INFO 09:54:32 INFO [loop_until]: kubectl --namespace=xlou top pods 09:54:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:54:32 INFO [loop_until]: OK (rc = 0) 09:54:32 DEBUG --- stdout --- 09:54:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 5Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5768Mi ds-cts-0 8m 457Mi ds-cts-1 9m 411Mi ds-cts-2 9m 425Mi ds-idrepo-0 7025m 9898Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12553Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10330m 4579Mi idm-58dc667486-h86fb 9168m 4226Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3390m 2693Mi 09:54:32 DEBUG --- stderr --- 09:54:32 DEBUG 09:54:34 INFO 09:54:34 INFO [loop_until]: kubectl --namespace=xlou top node 09:54:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:54:34 INFO [loop_until]: OK (rc = 0) 09:54:34 DEBUG --- stdout --- 09:54:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 88m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 67m 0% 6655Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 126m 0% 1549Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 72m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14658m 92% 5748Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 9448m 59% 5345Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 80m 0% 1349Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1090Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1101Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1076Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13168Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7125m 44% 10452Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 72m 0% 14347Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3539m 22% 3631Mi 6% 09:54:34 DEBUG --- stderr --- 09:54:34 DEBUG 09:55:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264124" 09:55:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264124" 09:55:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264124" 09:55:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264124" 09:55:24 INFO [http_cmd]: http status code OK 09:55:24 DEBUG --- status code --- 09:55:24 INFO [http_cmd]: http status code OK 09:55:24 INFO [http_cmd]: http status code OK 09:55:24 DEBUG http status code is 200 (expected 200) 09:55:24 DEBUG --- status code --- 09:55:24 DEBUG --- status code --- 09:55:24 DEBUG --- http response --- 09:55:24 DEBUG http status code is 200 (expected 200) 09:55:24 DEBUG http status code is 200 (expected 200) 09:55:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:24 DEBUG --- http response --- 09:55:24 DEBUG --- http response --- 09:55:24 DEBUG ----- output ----- 09:55:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:55:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:55:24 DEBUG { 09:55:24 DEBUG ----- output ----- 09:55:24 DEBUG ----- output ----- 09:55:24 DEBUG "status": "success", 09:55:24 DEBUG { 09:55:24 DEBUG { 09:55:24 DEBUG "data": { 09:55:24 DEBUG "status": "success", 09:55:24 DEBUG "status": "success", 09:55:24 DEBUG "resultType": "vector", 09:55:24 DEBUG "data": { 09:55:24 DEBUG "data": { 09:55:24 DEBUG "result": [ 09:55:24 DEBUG "resultType": "vector", 09:55:24 DEBUG "resultType": "vector", 09:55:24 DEBUG { 09:55:24 DEBUG "result": [ 09:55:24 DEBUG "result": [ 09:55:24 DEBUG "metric": { 09:55:24 DEBUG { 09:55:24 DEBUG { 09:55:24 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:55:24 DEBUG "metric": { 09:55:24 DEBUG "metric": { 09:55:24 DEBUG }, 09:55:24 DEBUG "pod": "ds-idrepo-0" 09:55:24 DEBUG "pod": "ds-idrepo-0" 09:55:24 DEBUG "value": [ 09:55:24 DEBUG }, 09:55:24 DEBUG }, 09:55:24 DEBUG 09:55:24 DEBUG "value": [ 09:55:24 DEBUG "value": [ 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG [...] 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG [...] 09:55:24 DEBUG [...] 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG }, 09:55:24 DEBUG "pod": "ds-cts-1" 09:55:24 DEBUG "pod": "ds-cts-1" 09:55:24 DEBUG "value": [ 09:55:24 DEBUG }, 09:55:24 DEBUG }, 09:55:24 DEBUG 1685264124, 09:55:24 DEBUG "value": [ 09:55:24 DEBUG "value": [ 09:55:24 DEBUG "0" 09:55:24 DEBUG 1685264124, 09:55:24 DEBUG 1685264124, 09:55:24 DEBUG ] 09:55:24 DEBUG "0" 09:55:24 DEBUG "0" 09:55:24 DEBUG } 09:55:24 DEBUG ] 09:55:24 DEBUG ] 09:55:24 DEBUG ] 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:24 DEBUG ] 09:55:24 DEBUG ] 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:24 INFO [http_cmd]: http status code OK 09:55:24 DEBUG --- status code --- 09:55:24 DEBUG http status code is 200 (expected 200) 09:55:24 DEBUG --- http response --- 09:55:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:24 DEBUG ----- output ----- 09:55:24 DEBUG { 09:55:24 DEBUG "status": "success", 09:55:24 DEBUG "data": { 09:55:24 DEBUG "resultType": "vector", 09:55:24 DEBUG "result": [ 09:55:24 DEBUG { 09:55:24 DEBUG "metric": { 09:55:24 DEBUG "pod": "ds-idrepo-0" 09:55:24 DEBUG }, 09:55:24 DEBUG "value": [ 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG [...] 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG "pod": "ds-idrepo-1" 09:55:24 DEBUG }, 09:55:24 DEBUG "value": [ 09:55:24 DEBUG 1685264124, 09:55:24 DEBUG "0" 09:55:24 DEBUG ] 09:55:24 DEBUG } 09:55:24 DEBUG ] 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264124" 09:55:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264124" 09:55:24 INFO [http_cmd]: http status code OK 09:55:24 DEBUG --- status code --- 09:55:24 INFO [http_cmd]: http status code OK 09:55:24 DEBUG http status code is 200 (expected 200) 09:55:24 DEBUG --- status code --- 09:55:24 DEBUG --- http response --- 09:55:24 DEBUG http status code is 200 (expected 200) 09:55:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:24 DEBUG --- http response --- 09:55:24 DEBUG ----- output ----- 09:55:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:55:24 DEBUG { 09:55:24 DEBUG ----- output ----- 09:55:24 DEBUG "status": "success", 09:55:24 DEBUG { 09:55:24 DEBUG "data": { 09:55:24 DEBUG "status": "success", 09:55:24 DEBUG "resultType": "vector", 09:55:24 DEBUG "data": { 09:55:24 DEBUG "result": [ 09:55:24 DEBUG "resultType": "vector", 09:55:24 DEBUG { 09:55:24 DEBUG "result": [ 09:55:24 DEBUG "metric": { 09:55:24 DEBUG { 09:55:24 DEBUG "pod": "ds-idrepo-0" 09:55:24 DEBUG "metric": { 09:55:24 DEBUG }, 09:55:24 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:55:24 DEBUG "value": [ 09:55:24 DEBUG }, 09:55:24 DEBUG 09:55:24 DEBUG "value": [ 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG [...] 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG [...] 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG 09:55:24 DEBUG "pod": "ds-idrepo-1" 09:55:24 DEBUG 09:55:24 DEBUG }, 09:55:24 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:55:24 DEBUG "value": [ 09:55:24 DEBUG }, 09:55:24 DEBUG 1685264124, 09:55:24 DEBUG "value": [ 09:55:24 DEBUG "0" 09:55:24 DEBUG 1685264124, 09:55:24 DEBUG ] 09:55:24 DEBUG "0" 09:55:24 DEBUG } 09:55:24 DEBUG ] 09:55:24 DEBUG ] 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:24 DEBUG ] 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:24 DEBUG } 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264125" 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264125" 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "pod": "ds-cts-2" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "pod": "ds-cts-1" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG "0" 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG ] 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG } 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG ] 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:55:25 DEBUG } 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG } 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "4814434304" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264125" 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "instance": "10.142.15.199:9100" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "instance": "10.142.15.212:9100" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "0.00033333333332545106" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264125" 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "pod": "ds-idrepo-0" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "pod": "ds-idrepo-1" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "0" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264125" 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "0.2" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264125" 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264125" 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264125" 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264125" 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "0" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "instance": "10.142.15.199:9100" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "instance": "10.142.15.212:9100" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "26268.966666666667" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:55:25 DEBUG }, 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG "value": [ 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:55:25 DEBUG [...] 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG 09:55:25 DEBUG { 09:55:25 DEBUG 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG 09:55:25 DEBUG "data": { 09:55:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG }, 09:55:25 DEBUG "result": [ 09:55:25 DEBUG "value": [ 09:55:25 DEBUG { 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "139.80953681264293" 09:55:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:55:25 DEBUG ] 09:55:25 DEBUG }, 09:55:25 DEBUG } 09:55:25 DEBUG "value": [ 09:55:25 DEBUG ] 09:55:25 DEBUG 09:55:25 DEBUG } 09:55:25 DEBUG 09:55:25 DEBUG } 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "0.006624863842714394" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264125" 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264125" 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "0" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "instance": "10.142.15.199:9100" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "instance": "10.142.15.212:9100" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "0.0022333333333335757" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264125" 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264125" 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "instance": "10.142.15.199:9100" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "instance": "10.142.15.212:9100" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "28163.233333333334" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "3.483809657780462" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264125" 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "0" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264125" 09:55:25 INFO [http_cmd]: http status code OK 09:55:25 DEBUG --- status code --- 09:55:25 DEBUG http status code is 200 (expected 200) 09:55:25 DEBUG --- http response --- 09:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:25 DEBUG ----- output ----- 09:55:25 DEBUG { 09:55:25 DEBUG "status": "success", 09:55:25 DEBUG "data": { 09:55:25 DEBUG "resultType": "vector", 09:55:25 DEBUG "result": [ 09:55:25 DEBUG { 09:55:25 DEBUG "metric": { 09:55:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG [...] 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG 09:55:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:55:25 DEBUG }, 09:55:25 DEBUG "value": [ 09:55:25 DEBUG 1685264125, 09:55:25 DEBUG "0" 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG ] 09:55:25 DEBUG } 09:55:25 DEBUG } 09:55:32 INFO 09:55:32 INFO [loop_until]: kubectl --namespace=xlou top pods 09:55:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:55:32 INFO [loop_until]: OK (rc = 0) 09:55:32 DEBUG --- stdout --- 09:55:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 6m 5768Mi ds-cts-0 9m 458Mi ds-cts-1 8m 411Mi ds-cts-2 9m 424Mi ds-idrepo-0 7501m 9898Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 8413m 4588Mi idm-58dc667486-h86fb 8662m 4236Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3077m 2696Mi 09:55:32 DEBUG --- stderr --- 09:55:32 DEBUG 09:55:34 INFO 09:55:34 INFO [loop_until]: kubectl --namespace=xlou top node 09:55:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:55:34 INFO [loop_until]: OK (rc = 0) 09:55:34 DEBUG --- stdout --- 09:55:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 91m 0% 6846Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 68m 0% 6650Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 118m 0% 1559Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 74m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 14508m 91% 5749Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 8612m 54% 5351Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 78m 0% 1353Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1089Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 59m 0% 1104Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 57m 0% 1077Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 65m 0% 13168Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6808m 42% 10439Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14335Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 2940m 18% 3636Mi 6% 09:55:34 DEBUG --- stderr --- 09:55:34 DEBUG 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:56:25 DEBUG { 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG { 09:56:25 DEBUG { 09:56:25 DEBUG { 09:56:25 DEBUG { 09:56:25 DEBUG "data": { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "data": { 09:56:25 DEBUG "data": { 09:56:25 DEBUG "data": { 09:56:25 DEBUG "result": [ 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG { 09:56:25 DEBUG "result": [ 09:56:25 DEBUG "result": [ 09:56:25 DEBUG "result": [ 09:56:25 DEBUG "result": [ 09:56:25 DEBUG "metric": { 09:56:25 DEBUG { 09:56:25 DEBUG { 09:56:25 DEBUG { 09:56:25 DEBUG { 09:56:25 DEBUG "pod": "ds-idrepo-0" 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG }, 09:56:25 DEBUG "pod": "ds-idrepo-0" 09:56:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:56:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:56:25 DEBUG "pod": "ds-idrepo-0" 09:56:25 DEBUG "value": [ 09:56:25 DEBUG }, 09:56:25 DEBUG }, 09:56:25 DEBUG }, 09:56:25 DEBUG }, 09:56:25 DEBUG 09:56:25 DEBUG "value": [ 09:56:25 DEBUG "value": [ 09:56:25 DEBUG "value": [ 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG [...] 09:56:25 DEBUG [...] 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "ds-idrepo-1" 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG }, 09:56:25 DEBUG "pod": "ds-cts-1" 09:56:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:56:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:56:25 DEBUG "pod": "ds-cts-1" 09:56:25 DEBUG "value": [ 09:56:25 DEBUG }, 09:56:25 DEBUG }, 09:56:25 DEBUG }, 09:56:25 DEBUG }, 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG "value": [ 09:56:25 DEBUG "value": [ 09:56:25 DEBUG "value": [ 09:56:25 DEBUG "0" 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG ] 09:56:25 DEBUG "0" 09:56:25 DEBUG "0" 09:56:25 DEBUG "0" 09:56:25 DEBUG "0" 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG ] 09:56:25 DEBUG ] 09:56:25 DEBUG ] 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG ] 09:56:25 DEBUG ] 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "ds-idrepo-0" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "ds-idrepo-1" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "ds-cts-2" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "ds-cts-1" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "instance": "10.142.15.199:9100" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "instance": "10.142.15.212:9100" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0.0003333333333406093" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "ds-idrepo-0" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "ds-idrepo-1" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "4816330752" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0.2" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "instance": "10.142.15.199:9100" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "instance": "10.142.15.212:9100" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "28285.366666666665" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0.006544182555641781" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "instance": "10.142.15.199:9100" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "instance": "10.142.15.212:9100" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0.00220000000000032" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264185" 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG { 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG "data": { 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:56:25 DEBUG "result": [ 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:56:25 DEBUG "data": { 09:56:25 DEBUG }, 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "value": [ 09:56:25 DEBUG "result": [ 09:56:25 DEBUG 09:56:25 DEBUG { 09:56:25 DEBUG 09:56:25 DEBUG "metric": { 09:56:25 DEBUG 09:56:25 DEBUG "instance": "10.142.15.199:9100" 09:56:25 DEBUG [...] 09:56:25 DEBUG }, 09:56:25 DEBUG 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:56:25 DEBUG 09:56:25 DEBUG }, 09:56:25 DEBUG [...] 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG 09:56:25 DEBUG "1.3150249385483865" 09:56:25 DEBUG 09:56:25 DEBUG ] 09:56:25 DEBUG "instance": "10.142.15.212:9100" 09:56:25 DEBUG } 09:56:25 DEBUG }, 09:56:25 DEBUG ] 09:56:25 DEBUG "value": [ 09:56:25 DEBUG } 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG } 09:56:25 DEBUG "28153.533333333333" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264185" 09:56:25 INFO [http_cmd]: http status code OK 09:56:25 DEBUG --- status code --- 09:56:25 DEBUG http status code is 200 (expected 200) 09:56:25 DEBUG --- http response --- 09:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:56:25 DEBUG ----- output ----- 09:56:25 DEBUG { 09:56:25 DEBUG "status": "success", 09:56:25 DEBUG "data": { 09:56:25 DEBUG "resultType": "vector", 09:56:25 DEBUG "result": [ 09:56:25 DEBUG { 09:56:25 DEBUG "metric": { 09:56:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG [...] 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG 09:56:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:56:25 DEBUG }, 09:56:25 DEBUG "value": [ 09:56:25 DEBUG 1685264185, 09:56:25 DEBUG "0" 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG ] 09:56:25 DEBUG } 09:56:25 DEBUG } 09:56:32 INFO 09:56:32 INFO [loop_until]: kubectl --namespace=xlou top pods 09:56:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:56:32 INFO [loop_until]: OK (rc = 0) 09:56:32 DEBUG --- stdout --- 09:56:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5768Mi ds-cts-0 8m 458Mi ds-cts-1 8m 411Mi ds-cts-2 8m 424Mi ds-idrepo-0 21m 9898Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 9m 4588Mi idm-58dc667486-h86fb 9m 4236Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 221m 350Mi 09:56:32 DEBUG --- stderr --- 09:56:32 DEBUG 09:56:34 INFO 09:56:34 INFO [loop_until]: kubectl --namespace=xlou top node 09:56:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:56:34 INFO [loop_until]: OK (rc = 0) 09:56:34 DEBUG --- stdout --- 09:56:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 88m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 67m 0% 6652Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 134m 0% 1552Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 74m 0% 6816Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 89m 0% 5749Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 84m 0% 5354Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 79m 0% 1351Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1088Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 55m 0% 1102Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13169Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 84m 0% 10445Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14336Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 67m 0% 1299Mi 2% 09:56:34 DEBUG --- stderr --- 09:56:34 DEBUG 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "pod": "ds-idrepo-0" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "ds-cts-1" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0" 09:57:25 DEBUG ] 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG } 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG ] 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG } 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG } 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "pod": "ds-idrepo-0" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "ds-cts-1" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG "data": { 09:57:25 DEBUG { 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG "result": [ 09:57:25 DEBUG "data": { 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:57:25 DEBUG { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "pod": "ds-idrepo-0" 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG }, 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "data": { 09:57:25 DEBUG "value": [ 09:57:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG 09:57:25 DEBUG }, 09:57:25 DEBUG "result": [ 09:57:25 DEBUG 09:57:25 DEBUG "value": [ 09:57:25 DEBUG { 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "metric": { 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG }, 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "ds-idrepo-1" 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG }, 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "value": [ 09:57:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:57:25 DEBUG [...] 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG }, 09:57:25 DEBUG 09:57:25 DEBUG "0" 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG ] 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG 09:57:25 DEBUG } 09:57:25 DEBUG "0" 09:57:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:57:25 DEBUG ] 09:57:25 DEBUG ] 09:57:25 DEBUG }, 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 DEBUG "value": [ 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 DEBUG "0" 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "pod": "ds-idrepo-0" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "ds-idrepo-1" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "instance": "10.142.15.199:9100" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "instance": "10.142.15.212:9100" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0.0003333333333330302" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "pod": "ds-cts-2" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "ds-cts-1" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG { 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:57:25 DEBUG "result": [ 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "pod": "ds-idrepo-0" 09:57:25 DEBUG "data": { 09:57:25 DEBUG }, 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "value": [ 09:57:25 DEBUG "result": [ 09:57:25 DEBUG 09:57:25 DEBUG { 09:57:25 DEBUG 09:57:25 DEBUG "metric": { 09:57:25 DEBUG 09:57:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:57:25 DEBUG [...] 09:57:25 DEBUG }, 09:57:25 DEBUG 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "ds-idrepo-1" 09:57:25 DEBUG 09:57:25 DEBUG }, 09:57:25 DEBUG [...] 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG 09:57:25 DEBUG "0" 09:57:25 DEBUG 09:57:25 DEBUG ] 09:57:25 DEBUG "pod": "idm-58dc667486-gclr8" 09:57:25 DEBUG } 09:57:25 DEBUG }, 09:57:25 DEBUG ] 09:57:25 DEBUG "value": [ 09:57:25 DEBUG } 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG } 09:57:25 DEBUG "0" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "4817551360" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0.2" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264245" 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "instance": "10.142.15.199:9100" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "instance": "10.142.15.212:9100" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "26373" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "idm-58dc667486-gclr8" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG { 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG "data": { 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "data": { 09:57:25 DEBUG "instance": "10.142.15.199:9100" 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG }, 09:57:25 DEBUG "result": [ 09:57:25 DEBUG "value": [ 09:57:25 DEBUG { 09:57:25 DEBUG 09:57:25 DEBUG "metric": { 09:57:25 DEBUG 09:57:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:57:25 DEBUG 09:57:25 DEBUG }, 09:57:25 DEBUG [...] 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "instance": "10.142.15.212:9100" 09:57:25 DEBUG [...] 09:57:25 DEBUG }, 09:57:25 DEBUG 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG 09:57:25 DEBUG "0.0027666666666666135" 09:57:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:57:25 DEBUG ] 09:57:25 DEBUG }, 09:57:25 DEBUG } 09:57:25 DEBUG "value": [ 09:57:25 DEBUG ] 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG } 09:57:25 DEBUG "0" 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264245" 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG { 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG "data": { 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "data": { 09:57:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG }, 09:57:25 DEBUG "result": [ 09:57:25 DEBUG "value": [ 09:57:25 DEBUG { 09:57:25 DEBUG 09:57:25 DEBUG "metric": { 09:57:25 DEBUG 09:57:25 DEBUG "instance": "10.142.15.199:9100" 09:57:25 DEBUG 09:57:25 DEBUG }, 09:57:25 DEBUG [...] 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:57:25 DEBUG [...] 09:57:25 DEBUG }, 09:57:25 DEBUG 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG 09:57:25 DEBUG "0.02666186800804979" 09:57:25 DEBUG "instance": "10.142.15.212:9100" 09:57:25 DEBUG ] 09:57:25 DEBUG }, 09:57:25 DEBUG } 09:57:25 DEBUG "value": [ 09:57:25 DEBUG ] 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG } 09:57:25 DEBUG "28027.533333333333" 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264245" 09:57:25 INFO [http_cmd]: http status code OK 09:57:25 DEBUG --- status code --- 09:57:25 DEBUG http status code is 200 (expected 200) 09:57:25 DEBUG --- http response --- 09:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:57:25 DEBUG ----- output ----- 09:57:25 DEBUG { 09:57:25 DEBUG "status": "success", 09:57:25 DEBUG "data": { 09:57:25 DEBUG "resultType": "vector", 09:57:25 DEBUG "result": [ 09:57:25 DEBUG { 09:57:25 DEBUG "metric": { 09:57:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG [...] 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG 09:57:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:57:25 DEBUG }, 09:57:25 DEBUG "value": [ 09:57:25 DEBUG 1685264245, 09:57:25 DEBUG "0" 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG ] 09:57:25 DEBUG } 09:57:25 DEBUG } 09:57:32 INFO 09:57:32 INFO [loop_until]: kubectl --namespace=xlou top pods 09:57:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:57:32 INFO [loop_until]: OK (rc = 0) 09:57:32 DEBUG --- stdout --- 09:57:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 7m 5768Mi ds-cts-0 8m 458Mi ds-cts-1 8m 411Mi ds-cts-2 9m 424Mi ds-idrepo-0 19m 9898Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 13m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 7m 4586Mi idm-58dc667486-h86fb 10m 4235Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 1m 350Mi 09:57:32 DEBUG --- stderr --- 09:57:32 DEBUG 09:57:35 INFO 09:57:35 INFO [loop_until]: kubectl --namespace=xlou top node 09:57:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:57:35 INFO [loop_until]: OK (rc = 0) 09:57:35 DEBUG --- stdout --- 09:57:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 90m 0% 6852Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 69m 0% 6652Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 118m 0% 1550Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 73m 0% 6816Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 83m 0% 5752Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 80m 0% 5351Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 78m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 58m 0% 1092Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 61m 0% 1115Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1076Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 60m 0% 13170Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 80m 0% 10442Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14334Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 67m 0% 1298Mi 2% 09:57:35 DEBUG --- stderr --- 09:57:35 DEBUG 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG { 09:58:25 DEBUG { 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "data": { 09:58:25 DEBUG "data": { 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG "result": [ 09:58:25 DEBUG "result": [ 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG { 09:58:25 DEBUG { 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:58:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:58:25 DEBUG "pod": "ds-idrepo-0" 09:58:25 DEBUG "pod": "ds-idrepo-0" 09:58:25 DEBUG }, 09:58:25 DEBUG }, 09:58:25 DEBUG }, 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG "value": [ 09:58:25 DEBUG "value": [ 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG [...] 09:58:25 DEBUG [...] 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:58:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:58:25 DEBUG "pod": "ds-cts-1" 09:58:25 DEBUG "pod": "ds-cts-1" 09:58:25 DEBUG }, 09:58:25 DEBUG }, 09:58:25 DEBUG }, 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG "value": [ 09:58:25 DEBUG "value": [ 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0.1" 09:58:25 DEBUG "0" 09:58:25 DEBUG "0" 09:58:25 DEBUG "0" 09:58:25 DEBUG ] 09:58:25 DEBUG ] 09:58:25 DEBUG ] 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG ] 09:58:25 DEBUG ] 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "instance": "10.142.15.199:9100" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "instance": "10.142.15.212:9100" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "ds-idrepo-0" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "ds-idrepo-1" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "ds-idrepo-0" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "ds-idrepo-1" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG "data": { 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG "result": [ 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG { 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:58:25 DEBUG "metric": { 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG "pod": "ds-idrepo-0" 09:58:25 DEBUG { 09:58:25 DEBUG }, 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "value": [ 09:58:25 DEBUG "data": { 09:58:25 DEBUG 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG 09:58:25 DEBUG "result": [ 09:58:25 DEBUG 09:58:25 DEBUG { 09:58:25 DEBUG [...] 09:58:25 DEBUG "metric": { 09:58:25 DEBUG 09:58:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:58:25 DEBUG 09:58:25 DEBUG }, 09:58:25 DEBUG 09:58:25 DEBUG "value": [ 09:58:25 DEBUG "pod": "ds-idrepo-1" 09:58:25 DEBUG 09:58:25 DEBUG }, 09:58:25 DEBUG 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG [...] 09:58:25 DEBUG "0" 09:58:25 DEBUG 09:58:25 DEBUG ] 09:58:25 DEBUG 09:58:25 DEBUG } 09:58:25 DEBUG 09:58:25 DEBUG ] 09:58:25 DEBUG "pod": "ds-idrepo-2" 09:58:25 DEBUG } 09:58:25 DEBUG }, 09:58:25 DEBUG } 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "ds-cts-2" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "ds-cts-1" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "4817043456" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0.2" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "ds-idrepo-2" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0.1" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "instance": "10.142.15.199:9100" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "instance": "10.142.15.212:9100" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "28408.333333333332" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "instance": "10.142.15.199:9100" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "instance": "10.142.15.212:9100" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0.0025333333333333505" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0.03333333333333333" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "instance": "10.142.15.199:9100" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "instance": "10.142.15.212:9100" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "28112.13333333333" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0.77666166139107" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264305" 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0.03333333333333333" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:25 INFO [http_cmd]: http status code OK 09:58:25 DEBUG --- status code --- 09:58:25 DEBUG http status code is 200 (expected 200) 09:58:25 DEBUG --- http response --- 09:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:58:25 DEBUG ----- output ----- 09:58:25 DEBUG { 09:58:25 DEBUG "status": "success", 09:58:25 DEBUG "data": { 09:58:25 DEBUG "resultType": "vector", 09:58:25 DEBUG "result": [ 09:58:25 DEBUG { 09:58:25 DEBUG "metric": { 09:58:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG [...] 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG 09:58:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 09:58:25 DEBUG }, 09:58:25 DEBUG "value": [ 09:58:25 DEBUG 1685264305, 09:58:25 DEBUG "0" 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG ] 09:58:25 DEBUG } 09:58:25 DEBUG } 09:58:32 INFO 09:58:32 INFO [loop_until]: kubectl --namespace=xlou top pods 09:58:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:58:32 INFO [loop_until]: OK (rc = 0) 09:58:32 DEBUG --- stdout --- 09:58:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 9m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 10m 458Mi ds-cts-1 9m 411Mi ds-cts-2 10m 424Mi ds-idrepo-0 2532m 9898Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 5529m 4594Mi idm-58dc667486-h86fb 3350m 4245Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3328m 2693Mi 09:58:32 DEBUG --- stderr --- 09:58:32 DEBUG 09:58:35 INFO 09:58:35 INFO [loop_until]: kubectl --namespace=xlou top node 09:58:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:58:35 INFO [loop_until]: OK (rc = 0) 09:58:35 DEBUG --- stdout --- 09:58:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 89m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 71m 0% 6651Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 135m 0% 1548Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 77m 0% 6818Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 7569m 47% 5793Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 5260m 33% 5366Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 81m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1090Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1103Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1076Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13171Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 4024m 25% 10441Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14333Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3814m 24% 3628Mi 6% 09:58:35 DEBUG --- stderr --- 09:58:35 DEBUG 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "ds-idrepo-0" 09:59:25 DEBUG "pod": "ds-idrepo-0" 09:59:25 DEBUG }, 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "ds-idrepo-1" 09:59:25 DEBUG "pod": "ds-cts-1" 09:59:25 DEBUG }, 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG "0" 09:59:25 DEBUG ] 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "ds-idrepo-0" 09:59:25 DEBUG }, 09:59:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:59:25 DEBUG 09:59:25 DEBUG "pod": "ds-cts-1" 09:59:25 DEBUG }, 09:59:25 DEBUG "pod": "ds-idrepo-2" 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264365" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG ] 09:59:25 DEBUG "0.41084937407457267" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "ds-idrepo-0" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "ds-idrepo-1" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "instance": "10.142.15.199:9100" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "instance": "10.142.15.212:9100" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 DEBUG "0.0003333333333330302" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "ds-idrepo-0" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "ds-idrepo-1" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "ds-idrepo-2" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "480777997.26692694" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "ds-cts-2" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "ds-cts-1" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "4840161280" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0.2" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "ds-idrepo-2" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0.005858800817762965" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "instance": "10.142.15.199:9100" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "instance": "10.142.15.212:9100" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "26406" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264365" 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:59:25 DEBUG "instance": "10.142.15.199:9100" 09:59:25 DEBUG }, 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:59:25 DEBUG "instance": "10.142.15.212:9100" 09:59:25 DEBUG }, 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG "0.0022666666666665944" 09:59:25 DEBUG ] 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264365" 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "instance": "10.142.15.199:9100" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "instance": "10.142.15.212:9100" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "27957.566666666666" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "3.619823056234281" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264365" 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:25 INFO [http_cmd]: http status code OK 09:59:25 DEBUG --- status code --- 09:59:25 DEBUG http status code is 200 (expected 200) 09:59:25 DEBUG --- http response --- 09:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 09:59:25 DEBUG ----- output ----- 09:59:25 DEBUG { 09:59:25 DEBUG "status": "success", 09:59:25 DEBUG "data": { 09:59:25 DEBUG "resultType": "vector", 09:59:25 DEBUG "result": [ 09:59:25 DEBUG { 09:59:25 DEBUG "metric": { 09:59:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG [...] 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG 09:59:25 DEBUG "pod": "ds-idrepo-2" 09:59:25 DEBUG }, 09:59:25 DEBUG "value": [ 09:59:25 DEBUG 1685264365, 09:59:25 DEBUG "0" 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG ] 09:59:25 DEBUG } 09:59:25 DEBUG } 09:59:32 INFO 09:59:32 INFO [loop_until]: kubectl --namespace=xlou top pods 09:59:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:59:32 INFO [loop_until]: OK (rc = 0) 09:59:32 DEBUG --- stdout --- 09:59:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 5Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5769Mi ds-cts-0 8m 458Mi ds-cts-1 9m 411Mi ds-cts-2 9m 425Mi ds-idrepo-0 7493m 9898Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 13m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11185m 4622Mi idm-58dc667486-h86fb 9942m 4265Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3826m 2697Mi 09:59:32 DEBUG --- stderr --- 09:59:32 DEBUG 09:59:35 INFO 09:59:35 INFO [loop_until]: kubectl --namespace=xlou top node 09:59:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:59:35 INFO [loop_until]: OK (rc = 0) 09:59:35 DEBUG --- stdout --- 09:59:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 87m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 66m 0% 6658Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 129m 0% 1550Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 74m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15852m 99% 5815Mi 9% gke-xlou-cdm-default-pool-f05840a3-b0st 10185m 64% 5376Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 84m 0% 1351Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1092Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1106Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1079Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13167Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7642m 48% 10445Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14336Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3774m 23% 3632Mi 6% 09:59:35 DEBUG --- stderr --- 09:59:35 DEBUG 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "ds-idrepo-0" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "ds-cts-1" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0.03333333333333333" 10:00:25 DEBUG ] 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264425" 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:00:25 DEBUG "pod": "ds-idrepo-0" 10:00:25 DEBUG }, 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:00:25 DEBUG "pod": "ds-idrepo-1" 10:00:25 DEBUG }, 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0" 10:00:25 DEBUG "0" 10:00:25 DEBUG ] 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG --- http response --- 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG { 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "data": { 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG "result": [ 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG { 10:00:25 DEBUG "data": { 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "ds-idrepo-0" 10:00:25 DEBUG "pod": "ds-idrepo-0" 10:00:25 DEBUG "result": [ 10:00:25 DEBUG "pod": "ds-idrepo-0" 10:00:25 DEBUG }, 10:00:25 DEBUG }, 10:00:25 DEBUG { 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG "value": [ 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "instance": "10.142.15.199:9100" 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG }, 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG "pod": "ds-idrepo-1" 10:00:25 DEBUG "pod": "ds-idrepo-1" 10:00:25 DEBUG 10:00:25 DEBUG "pod": "ds-cts-1" 10:00:25 DEBUG }, 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG }, 10:00:25 DEBUG 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "instance": "10.142.15.212:9100" 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0" 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG "0" 10:00:25 DEBUG }, 10:00:25 DEBUG "0" 10:00:25 DEBUG ] 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:00:25 DEBUG ] 10:00:25 DEBUG "value": [ 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG } 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG { 10:00:25 DEBUG ] 10:00:25 DEBUG "0.00033333333332545106" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 DEBUG "data": { 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG ] 10:00:25 DEBUG "result": [ 10:00:25 DEBUG } 10:00:25 DEBUG { 10:00:25 DEBUG } 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "ds-cts-2" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "4852604928" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "ds-cts-2" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "ds-cts-1" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0.2" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264425" 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "ds-cts-2" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "instance": "10.142.15.199:9100" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "instance": "10.142.15.212:9100" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "26403.63333333333" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG { 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG "data": { 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "data": { 10:00:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG }, 10:00:25 DEBUG "result": [ 10:00:25 DEBUG "value": [ 10:00:25 DEBUG { 10:00:25 DEBUG 10:00:25 DEBUG "metric": { 10:00:25 DEBUG 10:00:25 DEBUG "instance": "10.142.15.199:9100" 10:00:25 DEBUG 10:00:25 DEBUG }, 10:00:25 DEBUG [...] 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:00:25 DEBUG [...] 10:00:25 DEBUG }, 10:00:25 DEBUG 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG 10:00:25 DEBUG "0" 10:00:25 DEBUG "instance": "10.142.15.212:9100" 10:00:25 DEBUG ] 10:00:25 DEBUG }, 10:00:25 DEBUG } 10:00:25 DEBUG "value": [ 10:00:25 DEBUG ] 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG } 10:00:25 DEBUG "28132.266666666666" 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264425" 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:00:25 DEBUG "instance": "10.142.15.199:9100" 10:00:25 DEBUG }, 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:00:25 DEBUG "instance": "10.142.15.212:9100" 10:00:25 DEBUG }, 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0" 10:00:25 DEBUG "0.0021999999999998466" 10:00:25 DEBUG ] 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264425" 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "3.6988025154688144" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264425" 10:00:25 INFO [http_cmd]: http status code OK 10:00:25 DEBUG --- status code --- 10:00:25 DEBUG http status code is 200 (expected 200) 10:00:25 DEBUG --- http response --- 10:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:00:25 DEBUG ----- output ----- 10:00:25 DEBUG { 10:00:25 DEBUG "status": "success", 10:00:25 DEBUG "data": { 10:00:25 DEBUG "resultType": "vector", 10:00:25 DEBUG "result": [ 10:00:25 DEBUG { 10:00:25 DEBUG "metric": { 10:00:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG [...] 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG 10:00:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:00:25 DEBUG }, 10:00:25 DEBUG "value": [ 10:00:25 DEBUG 1685264425, 10:00:25 DEBUG "0" 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG ] 10:00:25 DEBUG } 10:00:25 DEBUG } 10:00:32 INFO 10:00:32 INFO [loop_until]: kubectl --namespace=xlou top pods 10:00:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:00:32 INFO [loop_until]: OK (rc = 0) 10:00:32 DEBUG --- stdout --- 10:00:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5768Mi ds-cts-0 8m 458Mi ds-cts-1 10m 411Mi ds-cts-2 9m 425Mi ds-idrepo-0 7632m 9899Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11303m 4649Mi idm-58dc667486-h86fb 9902m 4278Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3703m 2738Mi 10:00:32 DEBUG --- stderr --- 10:00:32 DEBUG 10:00:35 INFO 10:00:35 INFO [loop_until]: kubectl --namespace=xlou top node 10:00:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:00:35 INFO [loop_until]: OK (rc = 0) 10:00:35 DEBUG --- stdout --- 10:00:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 87m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6659Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 208m 1% 1590Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 75m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15426m 97% 5884Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 9830m 61% 5389Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 81m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 63m 0% 1093Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1100Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1079Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13169Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7678m 48% 10444Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 68m 0% 14336Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3737m 23% 3718Mi 6% 10:00:35 DEBUG --- stderr --- 10:00:35 DEBUG 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:25 DEBUG { 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG { 10:01:25 DEBUG "data": { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "result": [ 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG { 10:01:25 DEBUG "result": [ 10:01:25 DEBUG "metric": { 10:01:25 DEBUG { 10:01:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:01:25 DEBUG "metric": { 10:01:25 DEBUG }, 10:01:25 DEBUG "pod": "ds-idrepo-0" 10:01:25 DEBUG "value": [ 10:01:25 DEBUG }, 10:01:25 DEBUG 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "ds-idrepo-2" 10:01:25 DEBUG 10:01:25 DEBUG }, 10:01:25 DEBUG "pod": "ds-idrepo-1" 10:01:25 DEBUG "value": [ 10:01:25 DEBUG }, 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG "0.9689370190937587" 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG ] 10:01:25 DEBUG "0" 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "ds-idrepo-0" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "ds-cts-1" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:01:25 DEBUG "pod": "ds-idrepo-0" 10:01:25 DEBUG }, 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:01:25 DEBUG "pod": "ds-cts-1" 10:01:25 DEBUG }, 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0" 10:01:25 DEBUG "0" 10:01:25 DEBUG ] 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264485" 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "ds-idrepo-0" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "ds-idrepo-1" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "ds-idrepo-0" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "ds-idrepo-1" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "4862083072" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "instance": "10.142.15.199:9100" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "instance": "10.142.15.212:9100" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "ds-idrepo-2" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "400180367.28412646" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "ds-cts-2" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "ds-cts-1" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0.2" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264485" 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "instance": "10.142.15.199:9100" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "instance": "10.142.15.212:9100" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "28127.7" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:01:25 DEBUG { 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "data": { 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "result": [ 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG }, 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "instance": "10.142.15.199:9100" 10:01:25 DEBUG "value": [ 10:01:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:01:25 DEBUG }, 10:01:25 DEBUG 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG }, 10:01:25 DEBUG 10:01:25 DEBUG "instance": "10.142.15.212:9100" 10:01:25 DEBUG "value": [ 10:01:25 DEBUG "pod": "ds-idrepo-2" 10:01:25 DEBUG }, 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG "0" 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG ] 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "27797.633333333335" 10:01:25 DEBUG } 10:01:25 DEBUG "0.01415921823880799" 10:01:25 DEBUG ] 10:01:25 DEBUG ] 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264485" 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "instance": "10.142.15.199:9100" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "instance": "10.142.15.212:9100" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0.002233333333333339" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264485" 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "3.6160349450177605" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264485" 10:01:25 INFO [http_cmd]: http status code OK 10:01:25 DEBUG --- status code --- 10:01:25 DEBUG http status code is 200 (expected 200) 10:01:25 DEBUG --- http response --- 10:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:01:25 DEBUG ----- output ----- 10:01:25 DEBUG { 10:01:25 DEBUG "status": "success", 10:01:25 DEBUG "data": { 10:01:25 DEBUG "resultType": "vector", 10:01:25 DEBUG "result": [ 10:01:25 DEBUG { 10:01:25 DEBUG "metric": { 10:01:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG [...] 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG 10:01:25 DEBUG "pod": "ds-idrepo-2" 10:01:25 DEBUG }, 10:01:25 DEBUG "value": [ 10:01:25 DEBUG 1685264485, 10:01:25 DEBUG "0" 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG ] 10:01:25 DEBUG } 10:01:25 DEBUG } 10:01:32 INFO 10:01:32 INFO [loop_until]: kubectl --namespace=xlou top pods 10:01:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:01:33 INFO [loop_until]: OK (rc = 0) 10:01:33 DEBUG --- stdout --- 10:01:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5768Mi ds-cts-0 9m 459Mi ds-cts-1 11m 412Mi ds-cts-2 8m 424Mi ds-idrepo-0 7565m 9899Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11163m 4700Mi idm-58dc667486-h86fb 9926m 4292Mi lodemon-cfd7686d9-6dlt4 1m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3492m 2803Mi 10:01:33 DEBUG --- stderr --- 10:01:33 DEBUG 10:01:35 INFO 10:01:35 INFO [loop_until]: kubectl --namespace=xlou top node 10:01:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:01:35 INFO [loop_until]: OK (rc = 0) 10:01:35 DEBUG --- stdout --- 10:01:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 85m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 66m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 130m 0% 1532Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 72m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15919m 100% 5953Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 10230m 64% 5402Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 76m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1088Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1104Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 62m 0% 1079Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 62m 0% 13171Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7630m 48% 10443Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 68m 0% 14337Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3584m 22% 3733Mi 6% 10:01:35 DEBUG --- stderr --- 10:01:35 DEBUG 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "ds-idrepo-0" 10:02:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:02:25 DEBUG }, 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "ds-cts-1" 10:02:25 DEBUG "pod": "ds-idrepo-2" 10:02:25 DEBUG }, 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0" 10:02:25 DEBUG "0.5607179752066116" 10:02:25 DEBUG ] 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:02:25 DEBUG { 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG { 10:02:25 DEBUG "data": { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "result": [ 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG { 10:02:25 DEBUG "result": [ 10:02:25 DEBUG "metric": { 10:02:25 DEBUG { 10:02:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:02:25 DEBUG "metric": { 10:02:25 DEBUG }, 10:02:25 DEBUG "pod": "ds-idrepo-0" 10:02:25 DEBUG "value": [ 10:02:25 DEBUG }, 10:02:25 DEBUG 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:02:25 DEBUG 10:02:25 DEBUG }, 10:02:25 DEBUG "pod": "ds-cts-1" 10:02:25 DEBUG "value": [ 10:02:25 DEBUG }, 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG "0" 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG ] 10:02:25 DEBUG "0" 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "ds-idrepo-0" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "ds-idrepo-1" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG "0" 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG ] 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG } 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG ] 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:02:25 DEBUG } 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG } 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "instance": "10.142.15.199:9100" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "instance": "10.142.15.212:9100" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "ds-idrepo-0" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "ds-idrepo-1" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "4882788352" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "ds-idrepo-0" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "ds-idrepo-1" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "ds-idrepo-2" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "14673.366391184574" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "ds-cts-2" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "ds-cts-1" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0.2" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "instance": "10.142.15.199:9100" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "instance": "10.142.15.212:9100" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "28186.366666666665" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:02:25 DEBUG "instance": "10.142.15.199:9100" 10:02:25 DEBUG }, 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "ds-idrepo-2" 10:02:25 DEBUG "instance": "10.142.15.212:9100" 10:02:25 DEBUG }, 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0.007109069203460281" 10:02:25 DEBUG "26464.533333333333" 10:02:25 DEBUG ] 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "instance": "10.142.15.199:9100" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "instance": "10.142.15.212:9100" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0.002233333333333339" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "0" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264545" 10:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264545" 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG { 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG "data": { 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:02:25 DEBUG "result": [ 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:02:25 DEBUG "data": { 10:02:25 DEBUG }, 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "value": [ 10:02:25 DEBUG "result": [ 10:02:25 DEBUG 10:02:25 DEBUG { 10:02:25 DEBUG 10:02:25 DEBUG "metric": { 10:02:25 DEBUG 10:02:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:02:25 DEBUG [...] 10:02:25 DEBUG }, 10:02:25 DEBUG 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:02:25 DEBUG 10:02:25 DEBUG }, 10:02:25 DEBUG [...] 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "0" 10:02:25 DEBUG "pod": "ds-idrepo-2" 10:02:25 DEBUG ] 10:02:25 DEBUG }, 10:02:25 DEBUG } 10:02:25 DEBUG "value": [ 10:02:25 DEBUG ] 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG } 10:02:25 DEBUG "0" 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:25 INFO [http_cmd]: http status code OK 10:02:25 DEBUG --- status code --- 10:02:25 DEBUG http status code is 200 (expected 200) 10:02:25 DEBUG --- http response --- 10:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:02:25 DEBUG ----- output ----- 10:02:25 DEBUG { 10:02:25 DEBUG "status": "success", 10:02:25 DEBUG "data": { 10:02:25 DEBUG "resultType": "vector", 10:02:25 DEBUG "result": [ 10:02:25 DEBUG { 10:02:25 DEBUG "metric": { 10:02:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG [...] 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG 10:02:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:02:25 DEBUG }, 10:02:25 DEBUG "value": [ 10:02:25 DEBUG 1685264545, 10:02:25 DEBUG "3.557452711752851" 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG ] 10:02:25 DEBUG } 10:02:25 DEBUG } 10:02:33 INFO 10:02:33 INFO [loop_until]: kubectl --namespace=xlou top pods 10:02:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:02:33 INFO [loop_until]: OK (rc = 0) 10:02:33 DEBUG --- stdout --- 10:02:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 5m 5768Mi ds-cts-0 8m 458Mi ds-cts-1 9m 410Mi ds-cts-2 8m 425Mi ds-idrepo-0 7544m 9899Mi ds-idrepo-1 14m 13724Mi ds-idrepo-2 15m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11080m 4745Mi idm-58dc667486-h86fb 9820m 4305Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3506m 2820Mi 10:02:33 DEBUG --- stderr --- 10:02:33 DEBUG 10:02:35 INFO 10:02:35 INFO [loop_until]: kubectl --namespace=xlou top node 10:02:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:02:35 INFO [loop_until]: OK (rc = 0) 10:02:35 DEBUG --- stdout --- 10:02:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 87m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 66m 0% 6657Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 136m 0% 1540Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 71m 0% 6818Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15407m 96% 6019Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 9765m 61% 5415Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 84m 0% 1349Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1089Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 59m 0% 1107Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1078Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 66m 0% 13172Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7469m 47% 10444Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 64m 0% 14341Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3618m 22% 3749Mi 6% 10:02:35 DEBUG --- stderr --- 10:02:35 DEBUG 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG { 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "data": { 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG "result": [ 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG { 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "ds-idrepo-0" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:03:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:03:25 DEBUG }, 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG }, 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG "value": [ 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG { 10:03:25 DEBUG [...] 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG 10:03:25 DEBUG "data": { 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG "pod": "ds-cts-1" 10:03:25 DEBUG "result": [ 10:03:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:03:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG }, 10:03:25 DEBUG { 10:03:25 DEBUG }, 10:03:25 DEBUG }, 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:25 DEBUG "value": [ 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "value": [ 10:03:25 DEBUG "value": [ 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "pod": "ds-idrepo-0" 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG { 10:03:25 DEBUG "0" 10:03:25 DEBUG }, 10:03:25 DEBUG "0" 10:03:25 DEBUG "0" 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG ] 10:03:25 DEBUG "value": [ 10:03:25 DEBUG ] 10:03:25 DEBUG ] 10:03:25 DEBUG "data": { 10:03:25 DEBUG } 10:03:25 DEBUG 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG ] 10:03:25 DEBUG 10:03:25 DEBUG ] 10:03:25 DEBUG ] 10:03:25 DEBUG "result": [ 10:03:25 DEBUG } 10:03:25 DEBUG 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 DEBUG { 10:03:25 DEBUG } 10:03:25 DEBUG [...] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 DEBUG "metric": { 10:03:25 DEBUG 10:03:25 DEBUG "pod": "ds-idrepo-0" 10:03:25 DEBUG 10:03:25 DEBUG }, 10:03:25 DEBUG 10:03:25 DEBUG "value": [ 10:03:25 DEBUG "pod": "ds-cts-1" 10:03:25 DEBUG 10:03:25 DEBUG }, 10:03:25 DEBUG 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG [...] 10:03:25 DEBUG "0" 10:03:25 DEBUG 10:03:25 DEBUG ] 10:03:25 DEBUG 10:03:25 DEBUG } 10:03:25 DEBUG 10:03:25 DEBUG ] 10:03:25 DEBUG "pod": "ds-idrepo-1" 10:03:25 DEBUG } 10:03:25 DEBUG }, 10:03:25 DEBUG } 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "instance": "10.142.15.199:9100" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "instance": "10.142.15.212:9100" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0.0003333333333406093" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264605" 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "ds-idrepo-0" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "pod": "ds-idrepo-1" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "ds-idrepo-0" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "pod": "ds-idrepo-1" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "4899315712" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0.2" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "1079332.1414750442" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "ds-cts-2" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "pod": "ds-cts-1" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "instance": "10.142.15.199:9100" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "instance": "10.142.15.212:9100" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "28307.966666666667" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264605" 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG { 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "data": { 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG "result": [ 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG { 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "instance": "10.142.15.199:9100" 10:03:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:03:25 DEBUG "instance": "10.142.15.199:9100" 10:03:25 DEBUG }, 10:03:25 DEBUG }, 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG "value": [ 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG [...] 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "instance": "10.142.15.212:9100" 10:03:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:03:25 DEBUG "instance": "10.142.15.212:9100" 10:03:25 DEBUG }, 10:03:25 DEBUG }, 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG "value": [ 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "28374.6" 10:03:25 DEBUG "6.996369962222467" 10:03:25 DEBUG "0.0025000000000000946" 10:03:25 DEBUG ] 10:03:25 DEBUG ] 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG ] 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264605" 10:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264605" 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "0" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:25 INFO [http_cmd]: http status code OK 10:03:25 DEBUG --- status code --- 10:03:25 DEBUG http status code is 200 (expected 200) 10:03:25 DEBUG --- http response --- 10:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:03:25 DEBUG ----- output ----- 10:03:25 DEBUG { 10:03:25 DEBUG "status": "success", 10:03:25 DEBUG "data": { 10:03:25 DEBUG "resultType": "vector", 10:03:25 DEBUG "result": [ 10:03:25 DEBUG { 10:03:25 DEBUG "metric": { 10:03:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG [...] 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG 10:03:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:03:25 DEBUG }, 10:03:25 DEBUG "value": [ 10:03:25 DEBUG 1685264605, 10:03:25 DEBUG "3.536620524467328" 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG ] 10:03:25 DEBUG } 10:03:25 DEBUG } 10:03:33 INFO 10:03:33 INFO [loop_until]: kubectl --namespace=xlou top pods 10:03:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:03:33 INFO [loop_until]: OK (rc = 0) 10:03:33 DEBUG --- stdout --- 10:03:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 9m 5768Mi ds-cts-0 9m 458Mi ds-cts-1 8m 410Mi ds-cts-2 8m 426Mi ds-idrepo-0 7400m 9900Mi ds-idrepo-1 13m 13724Mi ds-idrepo-2 13m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11046m 4797Mi idm-58dc667486-h86fb 9813m 4319Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3530m 2848Mi 10:03:33 DEBUG --- stderr --- 10:03:33 DEBUG 10:03:35 INFO 10:03:35 INFO [loop_until]: kubectl --namespace=xlou top node 10:03:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:03:35 INFO [loop_until]: OK (rc = 0) 10:03:35 DEBUG --- stdout --- 10:03:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 88m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6651Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 135m 0% 1540Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 80m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15923m 100% 6097Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 10110m 63% 5428Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1353Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1095Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 66m 0% 1102Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1077Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 65m 0% 13171Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7461m 46% 10448Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14339Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3627m 22% 3775Mi 6% 10:03:35 DEBUG --- stderr --- 10:03:35 DEBUG 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "data": { 10:04:25 DEBUG "data": { 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG "result": [ 10:04:25 DEBUG "result": [ 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:04:25 DEBUG "pod": "ds-idrepo-0" 10:04:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:04:25 DEBUG "pod": "ds-idrepo-0" 10:04:25 DEBUG }, 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG }, 10:04:25 DEBUG }, 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG "value": [ 10:04:25 DEBUG "value": [ 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG [...] 10:04:25 DEBUG [...] 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG { 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "data": { 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "pod": "ds-cts-1" 10:04:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:04:25 DEBUG "pod": "ds-idrepo-1" 10:04:25 DEBUG }, 10:04:25 DEBUG "result": [ 10:04:25 DEBUG }, 10:04:25 DEBUG }, 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG { 10:04:25 DEBUG "value": [ 10:04:25 DEBUG "value": [ 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "metric": { 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "0" 10:04:25 DEBUG "pod": "ds-idrepo-0" 10:04:25 DEBUG "0" 10:04:25 DEBUG "0" 10:04:25 DEBUG "0" 10:04:25 DEBUG ] 10:04:25 DEBUG }, 10:04:25 DEBUG ] 10:04:25 DEBUG ] 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG "value": [ 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG 10:04:25 DEBUG ] 10:04:25 DEBUG ] 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "pod": "ds-cts-1" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "0" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264665" 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "instance": "10.142.15.199:9100" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "instance": "10.142.15.212:9100" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "0" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264665" 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "pod": "ds-idrepo-0" 10:04:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:04:25 DEBUG }, 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "pod": "ds-idrepo-1" 10:04:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:04:25 DEBUG }, 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "0" 10:04:25 DEBUG "0.2" 10:04:25 DEBUG ] 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "pod": "ds-idrepo-0" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "pod": "ds-idrepo-1" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "0" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "4912558080" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "260706.41414351508" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "pod": "ds-cts-2" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "pod": "ds-cts-1" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "0" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264665" 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "instance": "10.142.15.199:9100" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "instance": "10.142.15.212:9100" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "27972.333333333332" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "instance": "10.142.15.199:9100" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "instance": "10.142.15.212:9100" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "26295.73333333333" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "0" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264665" 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "data": { 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG "result": [ 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "instance": "10.142.15.199:9100" 10:04:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:04:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:04:25 DEBUG }, 10:04:25 DEBUG }, 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG "value": [ 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [...] 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:04:25 DEBUG 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG { 10:04:25 DEBUG 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG 10:04:25 DEBUG "instance": "10.142.15.212:9100" 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:04:25 DEBUG }, 10:04:25 DEBUG "data": { 10:04:25 DEBUG }, 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "value": [ 10:04:25 DEBUG { 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "result": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "0.002399999999999854" 10:04:25 DEBUG { 10:04:25 DEBUG "0" 10:04:25 DEBUG "data": { 10:04:25 DEBUG "3.775491473896699" 10:04:25 DEBUG ] 10:04:25 DEBUG "metric": { 10:04:25 DEBUG ] 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:04:25 DEBUG } 10:04:25 DEBUG "result": [ 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG }, 10:04:25 DEBUG ] 10:04:25 DEBUG { 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG "value": [ 10:04:25 DEBUG } 10:04:25 DEBUG "metric": { 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG 10:04:25 DEBUG } 10:04:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:04:25 DEBUG } 10:04:25 DEBUG 10:04:25 DEBUG }, 10:04:25 DEBUG 10:04:25 DEBUG "value": [ 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:04:25 DEBUG 10:04:25 DEBUG }, 10:04:25 DEBUG 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:04:25 DEBUG "0" 10:04:25 DEBUG }, 10:04:25 DEBUG ] 10:04:25 DEBUG "value": [ 10:04:25 DEBUG } 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG ] 10:04:25 DEBUG "0" 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264665" 10:04:25 INFO [http_cmd]: http status code OK 10:04:25 DEBUG --- status code --- 10:04:25 DEBUG http status code is 200 (expected 200) 10:04:25 DEBUG --- http response --- 10:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:04:25 DEBUG ----- output ----- 10:04:25 DEBUG { 10:04:25 DEBUG "status": "success", 10:04:25 DEBUG "data": { 10:04:25 DEBUG "resultType": "vector", 10:04:25 DEBUG "result": [ 10:04:25 DEBUG { 10:04:25 DEBUG "metric": { 10:04:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG [...] 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG 10:04:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:04:25 DEBUG }, 10:04:25 DEBUG "value": [ 10:04:25 DEBUG 1685264665, 10:04:25 DEBUG "3.593904658985301" 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG ] 10:04:25 DEBUG } 10:04:25 DEBUG } 10:04:33 INFO 10:04:33 INFO [loop_until]: kubectl --namespace=xlou top pods 10:04:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:04:33 INFO [loop_until]: OK (rc = 0) 10:04:33 DEBUG --- stdout --- 10:04:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 5Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5769Mi ds-cts-0 9m 459Mi ds-cts-1 9m 410Mi ds-cts-2 8m 426Mi ds-idrepo-0 7035m 9900Mi ds-idrepo-1 15m 13724Mi ds-idrepo-2 13m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 5251m 4825Mi idm-58dc667486-h86fb 14514m 4528Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3885m 3045Mi 10:04:33 DEBUG --- stderr --- 10:04:33 DEBUG 10:04:35 INFO 10:04:35 INFO [loop_until]: kubectl --namespace=xlou top node 10:04:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:04:35 INFO [loop_until]: OK (rc = 0) 10:04:35 DEBUG --- stdout --- 10:04:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 90m 0% 6852Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 69m 0% 6652Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 127m 0% 1543Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 82m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 8780m 55% 6406Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 14838m 93% 5628Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 88m 0% 1361Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1092Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 59m 0% 1102Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13175Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 6763m 42% 10449Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14339Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3839m 24% 3956Mi 6% 10:04:35 DEBUG --- stderr --- 10:04:35 DEBUG 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:05:25 DEBUG { 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG { 10:05:25 DEBUG { 10:05:25 DEBUG "data": { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "data": { 10:05:25 DEBUG "result": [ 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG { 10:05:25 DEBUG "result": [ 10:05:25 DEBUG "result": [ 10:05:25 DEBUG "metric": { 10:05:25 DEBUG { 10:05:25 DEBUG { 10:05:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG }, 10:05:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:05:25 DEBUG "pod": "ds-idrepo-0" 10:05:25 DEBUG "value": [ 10:05:25 DEBUG }, 10:05:25 DEBUG }, 10:05:25 DEBUG 10:05:25 DEBUG "value": [ 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG }, 10:05:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:05:25 DEBUG "pod": "ds-cts-1" 10:05:25 DEBUG "value": [ 10:05:25 DEBUG }, 10:05:25 DEBUG }, 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG "value": [ 10:05:25 DEBUG "0" 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG ] 10:05:25 DEBUG "0" 10:05:25 DEBUG "0" 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG ] 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "pod": "ds-idrepo-0" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "ds-cts-1" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "pod": "ds-idrepo-0" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "ds-idrepo-1" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264725" 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "instance": "10.142.15.199:9100" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "instance": "10.142.15.212:9100" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0.2" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "pod": "ds-idrepo-0" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "ds-idrepo-1" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "pod": "ds-idrepo-0" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "ds-idrepo-1" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "5178273792" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "499266.5344638404" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "pod": "ds-cts-2" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "ds-cts-1" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264725" 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "instance": "10.142.15.199:9100" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "instance": "10.142.15.212:9100" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "26308.566666666666" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "instance": "10.142.15.199:9100" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "instance": "10.142.15.212:9100" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "28017.8" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "instance": "10.142.15.199:9100" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "instance": "10.142.15.212:9100" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0.0021666666666665907" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:25 DEBUG { 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG { 10:05:25 DEBUG { 10:05:25 DEBUG "data": { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "data": { 10:05:25 DEBUG "result": [ 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG { 10:05:25 DEBUG "result": [ 10:05:25 DEBUG "result": [ 10:05:25 DEBUG "metric": { 10:05:25 DEBUG { 10:05:25 DEBUG { 10:05:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG }, 10:05:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:05:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG "value": [ 10:05:25 DEBUG }, 10:05:25 DEBUG }, 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG 10:05:25 DEBUG "value": [ 10:05:25 DEBUG "value": [ 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG [...] 10:05:25 DEBUG { 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "data": { 10:05:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG }, 10:05:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:05:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:05:25 DEBUG "result": [ 10:05:25 DEBUG "value": [ 10:05:25 DEBUG }, 10:05:25 DEBUG }, 10:05:25 DEBUG { 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG "value": [ 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "4.866510400334233" 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:05:25 DEBUG ] 10:05:25 DEBUG "0" 10:05:25 DEBUG "0" 10:05:25 DEBUG }, 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG ] 10:05:25 DEBUG "value": [ 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 DEBUG 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG ] 10:05:25 DEBUG 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 DEBUG 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "0" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264725" 10:05:25 INFO [http_cmd]: http status code OK 10:05:25 DEBUG --- status code --- 10:05:25 DEBUG http status code is 200 (expected 200) 10:05:25 DEBUG --- http response --- 10:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:05:25 DEBUG ----- output ----- 10:05:25 DEBUG { 10:05:25 DEBUG "status": "success", 10:05:25 DEBUG "data": { 10:05:25 DEBUG "resultType": "vector", 10:05:25 DEBUG "result": [ 10:05:25 DEBUG { 10:05:25 DEBUG "metric": { 10:05:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG [...] 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG 10:05:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:05:25 DEBUG }, 10:05:25 DEBUG "value": [ 10:05:25 DEBUG 1685264725, 10:05:25 DEBUG "4.1673620231782005" 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG ] 10:05:25 DEBUG } 10:05:25 DEBUG } 10:05:33 INFO 10:05:33 INFO [loop_until]: kubectl --namespace=xlou top pods 10:05:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:05:33 INFO [loop_until]: OK (rc = 0) 10:05:33 DEBUG --- stdout --- 10:05:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 6m 5769Mi ds-cts-0 9m 460Mi ds-cts-1 8m 411Mi ds-cts-2 9m 426Mi ds-idrepo-0 7735m 9900Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11197m 4871Mi idm-58dc667486-h86fb 10464m 4576Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3619m 3028Mi 10:05:33 DEBUG --- stderr --- 10:05:33 DEBUG 10:05:35 INFO 10:05:35 INFO [loop_until]: kubectl --namespace=xlou top node 10:05:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:05:35 INFO [loop_until]: OK (rc = 0) 10:05:35 DEBUG --- stdout --- 10:05:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 89m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 68m 0% 6650Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 134m 0% 1552Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 83m 0% 6817Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15912m 100% 6253Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 9626m 60% 5692Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1353Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1092Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1104Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13174Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7427m 46% 10450Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14340Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3648m 22% 3958Mi 6% 10:05:35 DEBUG --- stderr --- 10:05:35 DEBUG 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG { 10:06:25 DEBUG { 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "data": { 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 DEBUG "data": { 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG "result": [ 10:06:25 DEBUG "result": [ 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG { 10:06:25 DEBUG { 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:06:25 DEBUG "pod": "ds-idrepo-0" 10:06:25 DEBUG "pod": "ds-idrepo-0" 10:06:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:06:25 DEBUG }, 10:06:25 DEBUG }, 10:06:25 DEBUG }, 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG "value": [ 10:06:25 DEBUG "value": [ 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG [...] 10:06:25 DEBUG [...] 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:06:25 DEBUG "pod": "ds-cts-1" 10:06:25 DEBUG "pod": "ds-cts-1" 10:06:25 DEBUG "pod": "ds-cts-2" 10:06:25 DEBUG }, 10:06:25 DEBUG }, 10:06:25 DEBUG }, 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG "value": [ 10:06:25 DEBUG "value": [ 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG "0" 10:06:25 DEBUG "0" 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG ] 10:06:25 DEBUG ] 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG ] 10:06:25 DEBUG ] 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "ds-idrepo-0" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "ds-idrepo-1" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0.2" 10:06:25 DEBUG ] 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG } 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG ] 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG } 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG } 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "instance": "10.142.15.199:9100" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "instance": "10.142.15.212:9100" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0.00033333333332545106" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "ds-idrepo-0" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "ds-idrepo-1" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "ds-idrepo-0" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG "pod": "ds-idrepo-1" 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG }, 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG "value": [ 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG "0" 10:06:25 DEBUG { 10:06:25 DEBUG ] 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG } 10:06:25 DEBUG "data": { 10:06:25 DEBUG ] 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG } 10:06:25 DEBUG "result": [ 10:06:25 DEBUG } 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "5183295488" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "ds-cts-2" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "ds-cts-1" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "instance": "10.142.15.199:9100" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "instance": "10.142.15.212:9100" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "28554.7" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "instance": "10.142.15.199:9100" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "instance": "10.142.15.212:9100" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "28230.833333333336" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "instance": "10.142.15.199:9100" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "instance": "10.142.15.212:9100" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0.002399999999999854" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "ds-cts-2" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "0" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264785" 10:06:25 INFO [http_cmd]: http status code OK 10:06:25 DEBUG --- status code --- 10:06:25 DEBUG http status code is 200 (expected 200) 10:06:25 DEBUG --- http response --- 10:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:06:25 DEBUG ----- output ----- 10:06:25 DEBUG { 10:06:25 DEBUG "status": "success", 10:06:25 DEBUG "data": { 10:06:25 DEBUG "resultType": "vector", 10:06:25 DEBUG "result": [ 10:06:25 DEBUG { 10:06:25 DEBUG "metric": { 10:06:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG [...] 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG 10:06:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:06:25 DEBUG }, 10:06:25 DEBUG "value": [ 10:06:25 DEBUG 1685264785, 10:06:25 DEBUG "3.5267721916485804" 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG ] 10:06:25 DEBUG } 10:06:25 DEBUG } 10:06:33 INFO 10:06:33 INFO [loop_until]: kubectl --namespace=xlou top pods 10:06:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:06:33 INFO [loop_until]: OK (rc = 0) 10:06:33 DEBUG --- stdout --- 10:06:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 5Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5768Mi ds-cts-0 9m 460Mi ds-cts-1 9m 412Mi ds-cts-2 9m 426Mi ds-idrepo-0 7779m 9899Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11287m 4908Mi idm-58dc667486-h86fb 10790m 4607Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3451m 3024Mi 10:06:33 DEBUG --- stderr --- 10:06:33 DEBUG 10:06:35 INFO 10:06:35 INFO [loop_until]: kubectl --namespace=xlou top node 10:06:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:06:36 INFO [loop_until]: OK (rc = 0) 10:06:36 DEBUG --- stdout --- 10:06:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 87m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 74m 0% 6655Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 123m 0% 1543Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 83m 0% 6821Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15914m 100% 6127Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 11538m 72% 5709Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 80m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 63m 0% 1093Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 60m 0% 1102Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 62m 0% 1077Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 66m 0% 13174Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7912m 49% 10451Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14339Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3611m 22% 3958Mi 6% 10:06:36 DEBUG --- stderr --- 10:06:36 DEBUG 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:07:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:07:25 DEBUG }, 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:07:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:07:25 DEBUG }, 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "0" 10:07:25 DEBUG "0" 10:07:25 DEBUG ] 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "ds-idrepo-0" 10:07:25 DEBUG "pod": "ds-idrepo-0" 10:07:25 DEBUG }, 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "pod": "ds-cts-1" 10:07:25 DEBUG "pod": "ds-cts-1" 10:07:25 DEBUG }, 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "0" 10:07:25 DEBUG "0" 10:07:25 DEBUG ] 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "ds-idrepo-0" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "pod": "ds-idrepo-1" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "0" 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "ds-idrepo-0" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "pod": "ds-idrepo-1" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "0" 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264845" 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:07:25 DEBUG }, 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG "value": [ 10:07:25 DEBUG { 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "0.2" 10:07:25 DEBUG "data": { 10:07:25 DEBUG ] 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG } 10:07:25 DEBUG "result": [ 10:07:25 DEBUG ] 10:07:25 DEBUG { 10:07:25 DEBUG } 10:07:25 DEBUG "metric": { 10:07:25 DEBUG } 10:07:25 DEBUG "instance": "10.142.15.199:9100" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "instance": "10.142.15.212:9100" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "0" 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264845" 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "0" 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "ds-idrepo-0" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG [...] 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:07:25 DEBUG 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG 10:07:25 DEBUG { 10:07:25 DEBUG "pod": "ds-idrepo-1" 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG }, 10:07:25 DEBUG "data": { 10:07:25 DEBUG "value": [ 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "result": [ 10:07:25 DEBUG "0" 10:07:25 DEBUG { 10:07:25 DEBUG ] 10:07:25 DEBUG "metric": { 10:07:25 DEBUG } 10:07:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:07:25 DEBUG ] 10:07:25 DEBUG }, 10:07:25 DEBUG } 10:07:25 DEBUG "value": [ 10:07:25 DEBUG } 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "5218791424" 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "ds-cts-2" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "pod": "ds-cts-1" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "0" 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264845" 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "instance": "10.142.15.199:9100" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "instance": "10.142.15.212:9100" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "26507.366666666665" 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG { 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264845" 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264845" 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264845" 10:07:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:07:25 DEBUG }, 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:07:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:07:25 DEBUG }, 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "0" 10:07:25 DEBUG "0" 10:07:25 DEBUG ] 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG { 10:07:25 DEBUG { 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:07:25 DEBUG "instance": "10.142.15.199:9100" 10:07:25 DEBUG }, 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:07:25 DEBUG "instance": "10.142.15.212:9100" 10:07:25 DEBUG }, 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG "value": [ 10:07:25 INFO [http_cmd]: http status code OK 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG --- status code --- 10:07:25 DEBUG "0" 10:07:25 DEBUG "28251.333333333336" 10:07:25 DEBUG http status code is 200 (expected 200) 10:07:25 DEBUG ] 10:07:25 DEBUG ] 10:07:25 DEBUG --- http response --- 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:07:25 DEBUG ] 10:07:25 DEBUG ] 10:07:25 DEBUG ----- output ----- 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG { 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:25 DEBUG "status": "success", 10:07:25 DEBUG "data": { 10:07:25 DEBUG "resultType": "vector", 10:07:25 DEBUG "result": [ 10:07:25 DEBUG { 10:07:25 DEBUG "metric": { 10:07:25 DEBUG "instance": "10.142.15.199:9100" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG [...] 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG 10:07:25 DEBUG "instance": "10.142.15.212:9100" 10:07:25 DEBUG }, 10:07:25 DEBUG "value": [ 10:07:25 DEBUG 1685264845, 10:07:25 DEBUG "0.0024999999999996206" 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG ] 10:07:25 DEBUG } 10:07:25 DEBUG } 10:07:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264846" 10:07:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264846" 10:07:26 INFO [http_cmd]: http status code OK 10:07:26 DEBUG --- status code --- 10:07:26 DEBUG http status code is 200 (expected 200) 10:07:26 DEBUG --- http response --- 10:07:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:07:26 DEBUG ----- output ----- 10:07:26 DEBUG { 10:07:26 DEBUG "status": "success", 10:07:26 DEBUG "data": { 10:07:26 DEBUG "resultType": "vector", 10:07:26 DEBUG "result": [ 10:07:26 DEBUG { 10:07:26 DEBUG "metric": { 10:07:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:07:26 DEBUG }, 10:07:26 DEBUG "value": [ 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG [...] 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:07:26 DEBUG }, 10:07:26 DEBUG "value": [ 10:07:26 DEBUG 1685264846, 10:07:26 DEBUG "0" 10:07:26 DEBUG ] 10:07:26 DEBUG } 10:07:26 DEBUG ] 10:07:26 DEBUG } 10:07:26 DEBUG } 10:07:26 INFO [http_cmd]: http status code OK 10:07:26 DEBUG --- status code --- 10:07:26 DEBUG http status code is 200 (expected 200) 10:07:26 DEBUG --- http response --- 10:07:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:07:26 DEBUG ----- output ----- 10:07:26 DEBUG { 10:07:26 DEBUG "status": "success", 10:07:26 DEBUG "data": { 10:07:26 DEBUG "resultType": "vector", 10:07:26 DEBUG "result": [ 10:07:26 DEBUG { 10:07:26 DEBUG "metric": { 10:07:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:07:26 DEBUG }, 10:07:26 DEBUG "value": [ 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG [...] 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:07:26 DEBUG }, 10:07:26 DEBUG "value": [ 10:07:26 DEBUG 1685264846, 10:07:26 DEBUG "0" 10:07:26 DEBUG ] 10:07:26 DEBUG } 10:07:26 DEBUG ] 10:07:26 DEBUG } 10:07:26 DEBUG } 10:07:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264846" 10:07:26 INFO [http_cmd]: http status code OK 10:07:26 DEBUG --- status code --- 10:07:26 DEBUG http status code is 200 (expected 200) 10:07:26 DEBUG --- http response --- 10:07:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:07:26 DEBUG ----- output ----- 10:07:26 DEBUG { 10:07:26 DEBUG "status": "success", 10:07:26 DEBUG "data": { 10:07:26 DEBUG "resultType": "vector", 10:07:26 DEBUG "result": [ 10:07:26 DEBUG { 10:07:26 DEBUG "metric": { 10:07:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:07:26 DEBUG }, 10:07:26 DEBUG "value": [ 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG [...] 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG 10:07:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:07:26 DEBUG }, 10:07:26 DEBUG "value": [ 10:07:26 DEBUG 1685264846, 10:07:26 DEBUG "3.5717164316025287" 10:07:26 DEBUG ] 10:07:26 DEBUG } 10:07:26 DEBUG ] 10:07:26 DEBUG } 10:07:26 DEBUG } 10:07:33 INFO 10:07:33 INFO [loop_until]: kubectl --namespace=xlou top pods 10:07:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:07:33 INFO [loop_until]: OK (rc = 0) 10:07:33 DEBUG --- stdout --- 10:07:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 5m 5769Mi ds-cts-0 9m 460Mi ds-cts-1 8m 412Mi ds-cts-2 10m 427Mi ds-idrepo-0 7573m 9901Mi ds-idrepo-1 16m 13725Mi ds-idrepo-2 13m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11181m 4927Mi idm-58dc667486-h86fb 9717m 4614Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3441m 3023Mi 10:07:33 DEBUG --- stderr --- 10:07:33 DEBUG 10:07:36 INFO 10:07:36 INFO [loop_until]: kubectl --namespace=xlou top node 10:07:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:07:36 INFO [loop_until]: OK (rc = 0) 10:07:36 DEBUG --- stdout --- 10:07:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 97m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 74m 0% 6653Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 134m 0% 1542Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 80m 0% 6822Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15317m 96% 6091Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 10194m 64% 5716Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 81m 0% 1346Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1094Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1105Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1078Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 62m 0% 13172Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7436m 46% 10453Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 69m 0% 14338Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3622m 22% 3959Mi 6% 10:07:36 DEBUG --- stderr --- 10:07:36 DEBUG 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "0" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "0" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG { 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:08:25 DEBUG "data": { 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "pod": "ds-idrepo-0" 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG }, 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "value": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "pod": "ds-idrepo-0" 10:08:25 DEBUG 10:08:25 DEBUG "pod": "ds-idrepo-0" 10:08:25 DEBUG }, 10:08:25 DEBUG 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG [...] 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG "pod": "ds-idrepo-1" 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG }, 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG 10:08:25 DEBUG "pod": "ds-cts-1" 10:08:25 DEBUG "0" 10:08:25 DEBUG "pod": "ds-cts-1" 10:08:25 DEBUG }, 10:08:25 DEBUG ] 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG } 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG ] 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "0" 10:08:25 DEBUG } 10:08:25 DEBUG "0" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "pod": "ds-idrepo-0" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "pod": "ds-idrepo-1" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "0" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264905" 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "0.2" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "instance": "10.142.15.199:9100" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "instance": "10.142.15.212:9100" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "0.0003333333333406093" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264905" 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "550261.7660333511" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "pod": "ds-idrepo-0" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "pod": "ds-idrepo-1" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "0" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "5227302912" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:25 INFO [http_cmd]: http status code OK 10:08:25 DEBUG --- status code --- 10:08:25 DEBUG http status code is 200 (expected 200) 10:08:25 DEBUG --- http response --- 10:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:08:25 DEBUG ----- output ----- 10:08:25 DEBUG { 10:08:25 DEBUG "status": "success", 10:08:25 DEBUG "data": { 10:08:25 DEBUG "resultType": "vector", 10:08:25 DEBUG "result": [ 10:08:25 DEBUG { 10:08:25 DEBUG "metric": { 10:08:25 DEBUG "pod": "ds-cts-2" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG [...] 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG 10:08:25 DEBUG "pod": "ds-cts-1" 10:08:25 DEBUG }, 10:08:25 DEBUG "value": [ 10:08:25 DEBUG 1685264905, 10:08:25 DEBUG "0" 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG ] 10:08:25 DEBUG } 10:08:25 DEBUG } 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264905" 10:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264905" 10:08:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264905" 10:08:26 INFO [http_cmd]: http status code OK 10:08:26 INFO [http_cmd]: http status code OK 10:08:26 DEBUG --- status code --- 10:08:26 DEBUG --- status code --- 10:08:26 DEBUG http status code is 200 (expected 200) 10:08:26 DEBUG http status code is 200 (expected 200) 10:08:26 DEBUG --- http response --- 10:08:26 DEBUG --- http response --- 10:08:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:08:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:08:26 DEBUG ----- output ----- 10:08:26 DEBUG ----- output ----- 10:08:26 DEBUG { 10:08:26 DEBUG { 10:08:26 DEBUG "status": "success", 10:08:26 DEBUG "status": "success", 10:08:26 DEBUG "data": { 10:08:26 DEBUG "data": { 10:08:26 DEBUG "resultType": "vector", 10:08:26 DEBUG "resultType": "vector", 10:08:26 DEBUG "result": [ 10:08:26 DEBUG "result": [ 10:08:26 DEBUG { 10:08:26 DEBUG { 10:08:26 DEBUG "metric": { 10:08:26 DEBUG "metric": { 10:08:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:08:26 DEBUG "instance": "10.142.15.199:9100" 10:08:26 DEBUG }, 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG [...] 10:08:26 DEBUG [...] 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:08:26 DEBUG "instance": "10.142.15.212:9100" 10:08:26 DEBUG }, 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 1685264905, 10:08:26 DEBUG 1685264905, 10:08:26 DEBUG "0" 10:08:26 DEBUG "28227.76666666667" 10:08:26 DEBUG ] 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 DEBUG ] 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 INFO [http_cmd]: http status code OK 10:08:26 DEBUG --- status code --- 10:08:26 DEBUG http status code is 200 (expected 200) 10:08:26 DEBUG --- http response --- 10:08:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:08:26 DEBUG ----- output ----- 10:08:26 DEBUG { 10:08:26 DEBUG "status": "success", 10:08:26 DEBUG "data": { 10:08:26 DEBUG "resultType": "vector", 10:08:26 DEBUG "result": [ 10:08:26 DEBUG { 10:08:26 DEBUG "metric": { 10:08:26 DEBUG "instance": "10.142.15.199:9100" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG [...] 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG "instance": "10.142.15.212:9100" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 1685264905, 10:08:26 DEBUG "28097.333333333332" 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264906" 10:08:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264906" 10:08:26 INFO [http_cmd]: http status code OK 10:08:26 DEBUG --- status code --- 10:08:26 DEBUG http status code is 200 (expected 200) 10:08:26 DEBUG --- http response --- 10:08:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264906" 10:08:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:08:26 DEBUG ----- output ----- 10:08:26 DEBUG { 10:08:26 DEBUG "status": "success", 10:08:26 DEBUG "data": { 10:08:26 DEBUG "resultType": "vector", 10:08:26 DEBUG "result": [ 10:08:26 DEBUG { 10:08:26 DEBUG "metric": { 10:08:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG [...] 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 1685264906, 10:08:26 DEBUG "0" 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 INFO [http_cmd]: http status code OK 10:08:26 DEBUG --- status code --- 10:08:26 DEBUG http status code is 200 (expected 200) 10:08:26 DEBUG --- http response --- 10:08:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:08:26 DEBUG ----- output ----- 10:08:26 DEBUG { 10:08:26 DEBUG "status": "success", 10:08:26 DEBUG "data": { 10:08:26 DEBUG "resultType": "vector", 10:08:26 DEBUG "result": [ 10:08:26 DEBUG { 10:08:26 DEBUG "metric": { 10:08:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG [...] 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 1685264906, 10:08:26 DEBUG "0" 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 INFO [http_cmd]: http status code OK 10:08:26 DEBUG --- status code --- 10:08:26 DEBUG http status code is 200 (expected 200) 10:08:26 DEBUG --- http response --- 10:08:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:08:26 DEBUG ----- output ----- 10:08:26 DEBUG { 10:08:26 DEBUG "status": "success", 10:08:26 DEBUG "data": { 10:08:26 DEBUG "resultType": "vector", 10:08:26 DEBUG "result": [ 10:08:26 DEBUG { 10:08:26 DEBUG "metric": { 10:08:26 DEBUG "instance": "10.142.15.199:9100" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG [...] 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG "instance": "10.142.15.212:9100" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 1685264906, 10:08:26 DEBUG "0.002166666666666354" 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264906" 10:08:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264906" 10:08:26 INFO [http_cmd]: http status code OK 10:08:26 DEBUG --- status code --- 10:08:26 DEBUG http status code is 200 (expected 200) 10:08:26 DEBUG --- http response --- 10:08:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:08:26 DEBUG ----- output ----- 10:08:26 DEBUG { 10:08:26 DEBUG "status": "success", 10:08:26 DEBUG "data": { 10:08:26 DEBUG "resultType": "vector", 10:08:26 DEBUG "result": [ 10:08:26 DEBUG { 10:08:26 DEBUG "metric": { 10:08:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG [...] 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 1685264906, 10:08:26 DEBUG "0.00785872543368144" 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 INFO [http_cmd]: http status code OK 10:08:26 DEBUG --- status code --- 10:08:26 DEBUG http status code is 200 (expected 200) 10:08:26 DEBUG --- http response --- 10:08:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:08:26 DEBUG ----- output ----- 10:08:26 DEBUG { 10:08:26 DEBUG "status": "success", 10:08:26 DEBUG "data": { 10:08:26 DEBUG "resultType": "vector", 10:08:26 DEBUG "result": [ 10:08:26 DEBUG { 10:08:26 DEBUG "metric": { 10:08:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG [...] 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 1685264906, 10:08:26 DEBUG "0" 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264906" 10:08:26 INFO [http_cmd]: http status code OK 10:08:26 DEBUG --- status code --- 10:08:26 DEBUG http status code is 200 (expected 200) 10:08:26 DEBUG --- http response --- 10:08:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:08:26 DEBUG ----- output ----- 10:08:26 DEBUG { 10:08:26 DEBUG "status": "success", 10:08:26 DEBUG "data": { 10:08:26 DEBUG "resultType": "vector", 10:08:26 DEBUG "result": [ 10:08:26 DEBUG { 10:08:26 DEBUG "metric": { 10:08:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG [...] 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG 10:08:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:08:26 DEBUG }, 10:08:26 DEBUG "value": [ 10:08:26 DEBUG 1685264906, 10:08:26 DEBUG "3.51701916694766" 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG ] 10:08:26 DEBUG } 10:08:26 DEBUG } 10:08:33 INFO 10:08:33 INFO [loop_until]: kubectl --namespace=xlou top pods 10:08:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:08:33 INFO [loop_until]: OK (rc = 0) 10:08:33 DEBUG --- stdout --- 10:08:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 9m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 8m 412Mi ds-cts-2 9m 427Mi ds-idrepo-0 7492m 9901Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10918m 4966Mi idm-58dc667486-h86fb 9839m 4627Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3598m 3027Mi 10:08:33 DEBUG --- stderr --- 10:08:33 DEBUG 10:08:36 INFO 10:08:36 INFO [loop_until]: kubectl --namespace=xlou top node 10:08:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:08:36 INFO [loop_until]: OK (rc = 0) 10:08:36 DEBUG --- stdout --- 10:08:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 96m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6656Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 131m 0% 1543Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6824Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15924m 100% 6174Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 10161m 63% 5729Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 81m 0% 1347Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1094Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 55m 0% 1106Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 63m 0% 13176Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7718m 48% 10449Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14341Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3701m 23% 3958Mi 6% 10:08:36 DEBUG --- stderr --- 10:08:36 DEBUG 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG { 10:09:25 DEBUG { 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "data": { 10:09:25 DEBUG "data": { 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "result": [ 10:09:25 DEBUG "result": [ 10:09:25 DEBUG { 10:09:25 DEBUG { 10:09:25 DEBUG "metric": { 10:09:25 DEBUG "metric": { 10:09:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:09:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:09:25 DEBUG }, 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG [...] 10:09:25 DEBUG [...] 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:09:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:09:25 DEBUG }, 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG "0" 10:09:25 DEBUG "0" 10:09:25 DEBUG ] 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 DEBUG ] 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG { 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "data": { 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "result": [ 10:09:25 DEBUG { 10:09:25 DEBUG "metric": { 10:09:25 DEBUG "pod": "ds-idrepo-0" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG [...] 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG "pod": "ds-cts-1" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG "0" 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG { 10:09:25 DEBUG { 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "data": { 10:09:25 DEBUG "data": { 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "result": [ 10:09:25 DEBUG "result": [ 10:09:25 DEBUG { 10:09:25 DEBUG { 10:09:25 DEBUG "metric": { 10:09:25 DEBUG "metric": { 10:09:25 DEBUG "pod": "ds-idrepo-0" 10:09:25 DEBUG "pod": "ds-idrepo-0" 10:09:25 DEBUG }, 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG [...] 10:09:25 DEBUG [...] 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG "pod": "ds-idrepo-1" 10:09:25 DEBUG "pod": "ds-cts-1" 10:09:25 DEBUG }, 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG "0" 10:09:25 DEBUG "0" 10:09:25 DEBUG ] 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 DEBUG ] 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG { 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "data": { 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "result": [ 10:09:25 DEBUG { 10:09:25 DEBUG "metric": { 10:09:25 DEBUG "pod": "ds-idrepo-0" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG [...] 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG "pod": "ds-idrepo-1" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG "0" 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264965" 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG { 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "data": { 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "result": [ 10:09:25 DEBUG { 10:09:25 DEBUG "metric": { 10:09:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG [...] 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG "0.2" 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 DEBUG ] 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG } 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG } 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG { 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "data": { 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "result": [ 10:09:25 DEBUG { 10:09:25 DEBUG "metric": { 10:09:25 DEBUG "instance": "10.142.15.199:9100" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG [...] 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG "instance": "10.142.15.212:9100" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG "0.0006666666666660604" 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685264965" 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG { 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "data": { 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "result": [ 10:09:25 DEBUG { 10:09:25 DEBUG "metric": { 10:09:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG [...] 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:09:25 DEBUG }, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG "395127.7518356517" 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 DEBUG --- status code --- 10:09:25 INFO [http_cmd]: http status code OK 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG --- status code --- 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG http status code is 200 (expected 200) 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:25 DEBUG --- http response --- 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:09:25 DEBUG { 10:09:25 DEBUG ----- output ----- 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG { 10:09:25 DEBUG "data": { 10:09:25 DEBUG "status": "success", 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG "data": { 10:09:25 DEBUG "result": [ 10:09:25 DEBUG "resultType": "vector", 10:09:25 DEBUG { 10:09:25 DEBUG "result": [ 10:09:25 DEBUG "metric": { 10:09:25 DEBUG { 10:09:25 DEBUG "pod": "ds-idrepo-0" 10:09:25 DEBUG "metric": { 10:09:25 DEBUG }, 10:09:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:09:25 DEBUG "value": [ 10:09:25 DEBUG }, 10:09:25 DEBUG 10:09:25 DEBUG "value": [ 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG [...] 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG [...] 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG 10:09:25 DEBUG "pod": "ds-idrepo-1" 10:09:25 DEBUG 10:09:25 DEBUG }, 10:09:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:09:25 DEBUG "value": [ 10:09:25 DEBUG }, 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG "value": [ 10:09:25 DEBUG "0" 10:09:25 DEBUG 1685264965, 10:09:25 DEBUG ] 10:09:25 DEBUG "5243199488" 10:09:25 DEBUG } 10:09:25 DEBUG ] 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 DEBUG ] 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 DEBUG } 10:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264965" 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG --- status code --- 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG { 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG "data": { 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG "result": [ 10:09:26 DEBUG { 10:09:26 DEBUG "metric": { 10:09:26 DEBUG "pod": "ds-cts-2" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG "pod": "ds-cts-1" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 1685264965, 10:09:26 DEBUG "0" 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264966" 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG --- status code --- 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG { 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG "data": { 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG "result": [ 10:09:26 DEBUG { 10:09:26 DEBUG "metric": { 10:09:26 DEBUG "instance": "10.142.15.199:9100" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG "instance": "10.142.15.212:9100" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 1685264966, 10:09:26 DEBUG "27936.633333333335" 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264966" 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG --- status code --- 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG { 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG "data": { 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG "result": [ 10:09:26 DEBUG { 10:09:26 DEBUG "metric": { 10:09:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 1685264966, 10:09:26 DEBUG "0" 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264966" 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG --- status code --- 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG { 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG "data": { 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG "result": [ 10:09:26 DEBUG { 10:09:26 DEBUG "metric": { 10:09:26 DEBUG "instance": "10.142.15.199:9100" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG "instance": "10.142.15.212:9100" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 1685264966, 10:09:26 DEBUG "25766.13333333333" 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264966" 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG --- status code --- 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG { 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG "data": { 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG "result": [ 10:09:26 DEBUG { 10:09:26 DEBUG "metric": { 10:09:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 1685264966, 10:09:26 DEBUG "0" 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264966" 10:09:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685264966" 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG --- status code --- 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG { 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG "data": { 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG "result": [ 10:09:26 DEBUG { 10:09:26 DEBUG "metric": { 10:09:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 1685264966, 10:09:26 DEBUG "0" 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG --- status code --- 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG { 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG "data": { 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG "result": [ 10:09:26 DEBUG { 10:09:26 DEBUG "metric": { 10:09:26 DEBUG "instance": "10.142.15.199:9100" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG "instance": "10.142.15.212:9100" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 1685264966, 10:09:26 DEBUG "0.002066666666666587" 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264966" 10:09:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685264966" 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG --- status code --- 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG --- status code --- 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:09:26 DEBUG { 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG { 10:09:26 DEBUG "data": { 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG "data": { 10:09:26 DEBUG "result": [ 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG { 10:09:26 DEBUG "result": [ 10:09:26 DEBUG "metric": { 10:09:26 DEBUG { 10:09:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:09:26 DEBUG "metric": { 10:09:26 DEBUG }, 10:09:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:09:26 DEBUG "value": [ 10:09:26 DEBUG }, 10:09:26 DEBUG 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:09:26 DEBUG 10:09:26 DEBUG }, 10:09:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:09:26 DEBUG "value": [ 10:09:26 DEBUG }, 10:09:26 DEBUG 1685264966, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG "0" 10:09:26 DEBUG 1685264966, 10:09:26 DEBUG ] 10:09:26 DEBUG "6.905943584534398" 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685264966" 10:09:26 INFO [http_cmd]: http status code OK 10:09:26 DEBUG --- status code --- 10:09:26 DEBUG http status code is 200 (expected 200) 10:09:26 DEBUG --- http response --- 10:09:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:09:26 DEBUG ----- output ----- 10:09:26 DEBUG { 10:09:26 DEBUG "status": "success", 10:09:26 DEBUG "data": { 10:09:26 DEBUG "resultType": "vector", 10:09:26 DEBUG "result": [ 10:09:26 DEBUG { 10:09:26 DEBUG "metric": { 10:09:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG [...] 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG 10:09:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:09:26 DEBUG }, 10:09:26 DEBUG "value": [ 10:09:26 DEBUG 1685264966, 10:09:26 DEBUG "3.5774547983318716" 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG ] 10:09:26 DEBUG } 10:09:26 DEBUG } 10:09:33 INFO 10:09:33 INFO [loop_until]: kubectl --namespace=xlou top pods 10:09:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:09:33 INFO [loop_until]: OK (rc = 0) 10:09:33 DEBUG --- stdout --- 10:09:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 6m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 8m 412Mi ds-cts-2 8m 427Mi ds-idrepo-0 7647m 9901Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 13m 12555Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11037m 4992Mi idm-58dc667486-h86fb 9926m 4639Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3490m 3030Mi 10:09:33 DEBUG --- stderr --- 10:09:33 DEBUG 10:09:36 INFO 10:09:36 INFO [loop_until]: kubectl --namespace=xlou top node 10:09:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:09:36 INFO [loop_until]: OK (rc = 0) 10:09:36 DEBUG --- stdout --- 10:09:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 95m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 71m 0% 6658Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 133m 0% 1550Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15908m 100% 6231Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 10275m 64% 5738Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 79m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1095Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1105Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1076Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13175Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7790m 49% 10452Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14342Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3579m 22% 3957Mi 6% 10:09:36 DEBUG --- stderr --- 10:09:36 DEBUG 10:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265025" 10:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265025" 10:10:25 INFO [http_cmd]: http status code OK 10:10:25 DEBUG --- status code --- 10:10:25 DEBUG http status code is 200 (expected 200) 10:10:25 DEBUG --- http response --- 10:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:10:25 DEBUG ----- output ----- 10:10:25 DEBUG { 10:10:25 DEBUG "status": "success", 10:10:25 DEBUG "data": { 10:10:25 DEBUG "resultType": "vector", 10:10:25 DEBUG "result": [ 10:10:25 DEBUG { 10:10:25 DEBUG "metric": { 10:10:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG [...] 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG "pod": "idm-58dc667486-gclr8" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 1685265025, 10:10:25 DEBUG "0" 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG } 10:10:25 INFO [http_cmd]: http status code OK 10:10:25 DEBUG --- status code --- 10:10:25 DEBUG http status code is 200 (expected 200) 10:10:25 DEBUG --- http response --- 10:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:10:25 DEBUG ----- output ----- 10:10:25 DEBUG { 10:10:25 DEBUG "status": "success", 10:10:25 DEBUG "data": { 10:10:25 DEBUG "resultType": "vector", 10:10:25 DEBUG "result": [ 10:10:25 DEBUG { 10:10:25 DEBUG "metric": { 10:10:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG [...] 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 1685265025, 10:10:25 DEBUG "0" 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG } 10:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265025" 10:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265025" 10:10:25 INFO [http_cmd]: http status code OK 10:10:25 DEBUG --- status code --- 10:10:25 DEBUG http status code is 200 (expected 200) 10:10:25 DEBUG --- http response --- 10:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:10:25 DEBUG ----- output ----- 10:10:25 DEBUG { 10:10:25 DEBUG "status": "success", 10:10:25 DEBUG "data": { 10:10:25 DEBUG "resultType": "vector", 10:10:25 DEBUG "result": [ 10:10:25 DEBUG { 10:10:25 DEBUG "metric": { 10:10:25 DEBUG "pod": "ds-idrepo-0" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG [...] 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG "pod": "ds-cts-1" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 1685265025, 10:10:25 DEBUG "0" 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG } 10:10:25 INFO [http_cmd]: http status code OK 10:10:25 DEBUG --- status code --- 10:10:25 DEBUG http status code is 200 (expected 200) 10:10:25 DEBUG --- http response --- 10:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:10:25 DEBUG ----- output ----- 10:10:25 DEBUG { 10:10:25 DEBUG "status": "success", 10:10:25 DEBUG "data": { 10:10:25 DEBUG "resultType": "vector", 10:10:25 DEBUG "result": [ 10:10:25 DEBUG { 10:10:25 DEBUG "metric": { 10:10:25 DEBUG "pod": "ds-idrepo-0" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG [...] 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG "pod": "ds-idrepo-1" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 1685265025, 10:10:25 DEBUG "0" 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG } 10:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265025" 10:10:25 INFO [http_cmd]: http status code OK 10:10:25 DEBUG --- status code --- 10:10:25 DEBUG http status code is 200 (expected 200) 10:10:25 DEBUG --- http response --- 10:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:10:25 DEBUG ----- output ----- 10:10:25 DEBUG { 10:10:25 DEBUG "status": "success", 10:10:25 DEBUG "data": { 10:10:25 DEBUG "resultType": "vector", 10:10:25 DEBUG "result": [ 10:10:25 DEBUG { 10:10:25 DEBUG "metric": { 10:10:25 DEBUG "pod": "ds-idrepo-0" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265025" 10:10:25 DEBUG [...] 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG "pod": "ds-cts-1" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 1685265025, 10:10:25 DEBUG "0" 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG } 10:10:25 INFO [http_cmd]: http status code OK 10:10:25 DEBUG --- status code --- 10:10:25 DEBUG http status code is 200 (expected 200) 10:10:25 DEBUG --- http response --- 10:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:10:25 DEBUG ----- output ----- 10:10:25 DEBUG { 10:10:25 DEBUG "status": "success", 10:10:25 DEBUG "data": { 10:10:25 DEBUG "resultType": "vector", 10:10:25 DEBUG "result": [ 10:10:25 DEBUG { 10:10:25 DEBUG "metric": { 10:10:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG [...] 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 1685265025, 10:10:25 DEBUG "0.2" 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG } 10:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265025" 10:10:25 INFO [http_cmd]: http status code OK 10:10:25 DEBUG --- status code --- 10:10:25 DEBUG http status code is 200 (expected 200) 10:10:25 DEBUG --- http response --- 10:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:10:25 DEBUG ----- output ----- 10:10:25 DEBUG { 10:10:25 DEBUG "status": "success", 10:10:25 DEBUG "data": { 10:10:25 DEBUG "resultType": "vector", 10:10:25 DEBUG "result": [ 10:10:25 DEBUG { 10:10:25 DEBUG "metric": { 10:10:25 DEBUG "pod": "ds-idrepo-0" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG [...] 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG 10:10:25 DEBUG "pod": "ds-idrepo-1" 10:10:25 DEBUG }, 10:10:25 DEBUG "value": [ 10:10:25 DEBUG 1685265025, 10:10:25 DEBUG "0" 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG ] 10:10:25 DEBUG } 10:10:25 DEBUG } 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265026" 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "instance": "10.142.15.199:9100" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "instance": "10.142.15.212:9100" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "0.0006666666666660604" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265026" 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265026" 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265026" 10:10:26 DEBUG 10:10:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "129.35417653560714" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 DEBUG { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "pod": "ds-idrepo-0" 10:10:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:10:26 DEBUG }, 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG [...] 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "pod": "ds-idrepo-1" 10:10:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:10:26 DEBUG }, 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "0" 10:10:26 DEBUG "5251993600" 10:10:26 DEBUG ] 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265026" 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "pod": "ds-cts-2" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "pod": "ds-cts-1" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "0" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265026" 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265026" 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265026" 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265026" 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "instance": "10.142.15.199:9100" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "instance": "10.142.15.212:9100" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "28073.966666666667" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "status": "success", 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG "data": { 10:10:26 DEBUG "data": { 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG "result": [ 10:10:26 DEBUG "result": [ 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG { 10:10:26 DEBUG { 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:10:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:10:26 DEBUG { 10:10:26 DEBUG }, 10:10:26 DEBUG }, 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "value": [ 10:10:26 DEBUG "value": [ 10:10:26 DEBUG "data": { 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "result": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG { 10:10:26 DEBUG [...] 10:10:26 DEBUG [...] 10:10:26 DEBUG "metric": { 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "instance": "10.142.15.199:9100" 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG }, 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "value": [ 10:10:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:10:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:10:26 DEBUG 10:10:26 DEBUG }, 10:10:26 DEBUG }, 10:10:26 DEBUG 10:10:26 DEBUG "value": [ 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG [...] 10:10:26 DEBUG "0" 10:10:26 DEBUG "0" 10:10:26 DEBUG 10:10:26 DEBUG ] 10:10:26 DEBUG ] 10:10:26 DEBUG 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 DEBUG 10:10:26 DEBUG ] 10:10:26 DEBUG ] 10:10:26 DEBUG "instance": "10.142.15.212:9100" 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 DEBUG }, 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "26531.9" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265026" 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265026" 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "instance": "10.142.15.199:9100" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "instance": "10.142.15.212:9100" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "0.0021666666666670647" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "0" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265026" 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265026" 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "0.007150262403285574" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "0" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265026" 10:10:26 INFO [http_cmd]: http status code OK 10:10:26 DEBUG --- status code --- 10:10:26 DEBUG http status code is 200 (expected 200) 10:10:26 DEBUG --- http response --- 10:10:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:10:26 DEBUG ----- output ----- 10:10:26 DEBUG { 10:10:26 DEBUG "status": "success", 10:10:26 DEBUG "data": { 10:10:26 DEBUG "resultType": "vector", 10:10:26 DEBUG "result": [ 10:10:26 DEBUG { 10:10:26 DEBUG "metric": { 10:10:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG [...] 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG 10:10:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:10:26 DEBUG }, 10:10:26 DEBUG "value": [ 10:10:26 DEBUG 1685265026, 10:10:26 DEBUG "3.4817469337514524" 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG ] 10:10:26 DEBUG } 10:10:26 DEBUG } 10:10:33 INFO 10:10:33 INFO [loop_until]: kubectl --namespace=xlou top pods 10:10:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:10:34 INFO [loop_until]: OK (rc = 0) 10:10:34 DEBUG --- stdout --- 10:10:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 6m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 8m 412Mi ds-cts-2 8m 427Mi ds-idrepo-0 7395m 9901Mi ds-idrepo-1 13m 13725Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11246m 5034Mi idm-58dc667486-h86fb 9969m 4649Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3621m 3033Mi 10:10:34 DEBUG --- stderr --- 10:10:34 DEBUG 10:10:36 INFO 10:10:36 INFO [loop_until]: kubectl --namespace=xlou top node 10:10:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:10:36 INFO [loop_until]: OK (rc = 0) 10:10:36 DEBUG --- stdout --- 10:10:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 97m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 73m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 129m 0% 1547Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6826Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15894m 100% 6314Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 9824m 61% 5744Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1351Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1098Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1105Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 65m 0% 13176Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7656m 48% 10451Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14344Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3750m 23% 3956Mi 6% 10:10:36 DEBUG --- stderr --- 10:10:36 DEBUG 10:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265085" 10:11:25 INFO [http_cmd]: http status code OK 10:11:25 DEBUG --- status code --- 10:11:25 DEBUG http status code is 200 (expected 200) 10:11:25 DEBUG --- http response --- 10:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:11:25 DEBUG ----- output ----- 10:11:25 DEBUG { 10:11:25 DEBUG "status": "success", 10:11:25 DEBUG "data": { 10:11:25 DEBUG "resultType": "vector", 10:11:25 DEBUG "result": [ 10:11:25 DEBUG { 10:11:25 DEBUG "metric": { 10:11:25 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:11:25 DEBUG }, 10:11:25 DEBUG "value": [ 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG [...] 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:11:25 DEBUG }, 10:11:25 DEBUG "value": [ 10:11:25 DEBUG 1685265085, 10:11:25 DEBUG "0" 10:11:25 DEBUG ] 10:11:25 DEBUG } 10:11:25 DEBUG ] 10:11:25 DEBUG } 10:11:25 DEBUG } 10:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265085" 10:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265085" 10:11:25 INFO [http_cmd]: http status code OK 10:11:25 DEBUG --- status code --- 10:11:25 DEBUG http status code is 200 (expected 200) 10:11:25 DEBUG --- http response --- 10:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:11:25 DEBUG ----- output ----- 10:11:25 DEBUG { 10:11:25 DEBUG "status": "success", 10:11:25 DEBUG "data": { 10:11:25 DEBUG "resultType": "vector", 10:11:25 DEBUG "result": [ 10:11:25 DEBUG { 10:11:25 DEBUG "metric": { 10:11:25 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:11:25 DEBUG }, 10:11:25 DEBUG "value": [ 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG [...] 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:11:25 DEBUG }, 10:11:25 DEBUG "value": [ 10:11:25 DEBUG 1685265085, 10:11:25 DEBUG "0" 10:11:25 DEBUG ] 10:11:25 DEBUG } 10:11:25 DEBUG ] 10:11:25 DEBUG } 10:11:25 DEBUG } 10:11:25 INFO [http_cmd]: http status code OK 10:11:25 DEBUG --- status code --- 10:11:25 DEBUG http status code is 200 (expected 200) 10:11:25 DEBUG --- http response --- 10:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:11:25 DEBUG ----- output ----- 10:11:25 DEBUG { 10:11:25 DEBUG "status": "success", 10:11:25 DEBUG "data": { 10:11:25 DEBUG "resultType": "vector", 10:11:25 DEBUG "result": [ 10:11:25 DEBUG { 10:11:25 DEBUG "metric": { 10:11:25 DEBUG "pod": "ds-idrepo-0" 10:11:25 DEBUG }, 10:11:25 DEBUG "value": [ 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG [...] 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG "pod": "ds-cts-1" 10:11:25 DEBUG }, 10:11:25 DEBUG "value": [ 10:11:25 DEBUG 1685265085, 10:11:25 DEBUG "0" 10:11:25 DEBUG ] 10:11:25 DEBUG } 10:11:25 DEBUG ] 10:11:25 DEBUG } 10:11:25 DEBUG } 10:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265085" 10:11:25 INFO [http_cmd]: http status code OK 10:11:25 DEBUG --- status code --- 10:11:25 DEBUG http status code is 200 (expected 200) 10:11:25 DEBUG --- http response --- 10:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:11:25 DEBUG ----- output ----- 10:11:25 DEBUG { 10:11:25 DEBUG "status": "success", 10:11:25 DEBUG "data": { 10:11:25 DEBUG "resultType": "vector", 10:11:25 DEBUG "result": [ 10:11:25 DEBUG { 10:11:25 DEBUG "metric": { 10:11:25 DEBUG "pod": "ds-idrepo-0" 10:11:25 DEBUG }, 10:11:25 DEBUG "value": [ 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG [...] 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG 10:11:25 DEBUG "pod": "ds-idrepo-1" 10:11:25 DEBUG }, 10:11:25 DEBUG "value": [ 10:11:25 DEBUG 1685265085, 10:11:25 DEBUG "0" 10:11:25 DEBUG ] 10:11:25 DEBUG } 10:11:25 DEBUG ] 10:11:25 DEBUG } 10:11:25 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "ds-idrepo-0" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "ds-cts-1" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0.2" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "ds-idrepo-0" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "ds-idrepo-1" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "ds-idrepo-0" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "ds-idrepo-1" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "ds-cts-2" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "13828553.842366047" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "instance": "10.142.15.199:9100" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "instance": "10.142.15.212:9100" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "5264375808" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "ds-cts-2" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "ds-cts-1" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265086" 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "instance": "10.142.15.199:9100" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "instance": "10.142.15.212:9100" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "28635.433333333334" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "ds-cts-2" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG "metric": { 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG }, 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG "value": [ 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:11:26 DEBUG 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG 10:11:26 DEBUG { 10:11:26 DEBUG 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG [...] 10:11:26 DEBUG "data": { 10:11:26 DEBUG 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG 10:11:26 DEBUG "result": [ 10:11:26 DEBUG 10:11:26 DEBUG { 10:11:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:11:26 DEBUG "metric": { 10:11:26 DEBUG }, 10:11:26 DEBUG "instance": "10.142.15.199:9100" 10:11:26 DEBUG "value": [ 10:11:26 DEBUG }, 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG "0" 10:11:26 DEBUG 10:11:26 DEBUG ] 10:11:26 DEBUG 10:11:26 DEBUG } 10:11:26 DEBUG 10:11:26 DEBUG ] 10:11:26 DEBUG [...] 10:11:26 DEBUG } 10:11:26 DEBUG 10:11:26 DEBUG } 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "instance": "10.142.15.212:9100" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "28759.23333333333" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "instance": "10.142.15.199:9100" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "instance": "10.142.15.212:9100" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0.0020333333333335682" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "ds-cts-2" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0.007658914350782792" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "0" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265086" 10:11:26 INFO [http_cmd]: http status code OK 10:11:26 DEBUG --- status code --- 10:11:26 DEBUG http status code is 200 (expected 200) 10:11:26 DEBUG --- http response --- 10:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:11:26 DEBUG ----- output ----- 10:11:26 DEBUG { 10:11:26 DEBUG "status": "success", 10:11:26 DEBUG "data": { 10:11:26 DEBUG "resultType": "vector", 10:11:26 DEBUG "result": [ 10:11:26 DEBUG { 10:11:26 DEBUG "metric": { 10:11:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG [...] 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG 10:11:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:11:26 DEBUG }, 10:11:26 DEBUG "value": [ 10:11:26 DEBUG 1685265086, 10:11:26 DEBUG "3.886475982663539" 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG ] 10:11:26 DEBUG } 10:11:26 DEBUG } 10:11:34 INFO 10:11:34 INFO [loop_until]: kubectl --namespace=xlou top pods 10:11:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:11:34 INFO [loop_until]: OK (rc = 0) 10:11:34 DEBUG --- stdout --- 10:11:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 8m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 8m 412Mi ds-cts-2 8m 427Mi ds-idrepo-0 7783m 9901Mi ds-idrepo-1 19m 13725Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11011m 5154Mi idm-58dc667486-h86fb 9706m 4659Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3468m 3039Mi 10:11:34 DEBUG --- stderr --- 10:11:34 DEBUG 10:11:36 INFO 10:11:36 INFO [loop_until]: kubectl --namespace=xlou top node 10:11:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:11:36 INFO [loop_until]: OK (rc = 0) 10:11:36 DEBUG --- stdout --- 10:11:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 96m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 75m 0% 6659Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 130m 0% 1547Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 82m 0% 6823Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15953m 100% 6443Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 9851m 61% 5757Mi 9% gke-xlou-cdm-default-pool-f05840a3-bqd0 87m 0% 1349Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1095Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1105Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1077Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13175Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7347m 46% 10452Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 71m 0% 14342Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3589m 22% 3958Mi 6% 10:11:36 DEBUG --- stderr --- 10:11:36 DEBUG 10:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265145" 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265145, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "ds-idrepo-0" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "ds-cts-1" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "ds-idrepo-0" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "ds-idrepo-1" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "ds-idrepo-0" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "ds-cts-1" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0.2" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "ds-idrepo-0" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "ds-idrepo-1" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "ds-idrepo-0" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "ds-idrepo-1" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "ds-cts-2" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "14500635.750476008" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "instance": "10.142.15.199:9100" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "instance": "10.142.15.212:9100" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "5275455488" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "ds-cts-2" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "ds-cts-1" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265146" 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "instance": "10.142.15.199:9100" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "instance": "10.142.15.212:9100" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "28578.366666666665" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "ds-cts-2" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG { 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:12:26 DEBUG "data": { 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG { 10:12:26 DEBUG "result": [ 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG { 10:12:26 DEBUG "data": { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:12:26 DEBUG "result": [ 10:12:26 DEBUG }, 10:12:26 DEBUG { 10:12:26 DEBUG "value": [ 10:12:26 DEBUG "metric": { 10:12:26 DEBUG 10:12:26 DEBUG "instance": "10.142.15.199:9100" 10:12:26 DEBUG 10:12:26 DEBUG }, 10:12:26 DEBUG 10:12:26 DEBUG "value": [ 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:12:26 DEBUG 10:12:26 DEBUG }, 10:12:26 DEBUG 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "instance": "10.142.15.212:9100" 10:12:26 DEBUG "0" 10:12:26 DEBUG }, 10:12:26 DEBUG ] 10:12:26 DEBUG "value": [ 10:12:26 DEBUG } 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG ] 10:12:26 DEBUG "26775.699999999997" 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265146" 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "instance": "10.142.15.199:9100" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "instance": "10.142.15.212:9100" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0.0027666666666666135" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "pod": "ds-cts-2" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "0.007870169813146656" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265146" 10:12:26 INFO [http_cmd]: http status code OK 10:12:26 DEBUG --- status code --- 10:12:26 DEBUG http status code is 200 (expected 200) 10:12:26 DEBUG --- http response --- 10:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:12:26 DEBUG ----- output ----- 10:12:26 DEBUG { 10:12:26 DEBUG "status": "success", 10:12:26 DEBUG "data": { 10:12:26 DEBUG "resultType": "vector", 10:12:26 DEBUG "result": [ 10:12:26 DEBUG { 10:12:26 DEBUG "metric": { 10:12:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG [...] 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG 10:12:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:12:26 DEBUG }, 10:12:26 DEBUG "value": [ 10:12:26 DEBUG 1685265146, 10:12:26 DEBUG "3.592852026637066" 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG ] 10:12:26 DEBUG } 10:12:26 DEBUG } 10:12:34 INFO 10:12:34 INFO [loop_until]: kubectl --namespace=xlou top pods 10:12:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:12:34 INFO [loop_until]: OK (rc = 0) 10:12:34 DEBUG --- stdout --- 10:12:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 9m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 8m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 8m 412Mi ds-cts-2 9m 427Mi ds-idrepo-0 6030m 9901Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 6091m 5128Mi idm-58dc667486-h86fb 14092m 4852Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3667m 3061Mi 10:12:34 DEBUG --- stderr --- 10:12:34 DEBUG 10:12:36 INFO 10:12:36 INFO [loop_until]: kubectl --namespace=xlou top node 10:12:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:12:36 INFO [loop_until]: OK (rc = 0) 10:12:36 DEBUG --- stdout --- 10:12:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 96m 0% 6860Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 75m 0% 6669Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 129m 0% 1552Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 10045m 63% 6708Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 14838m 93% 5944Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 82m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 62m 0% 1092Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1105Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1078Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13178Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7033m 44% 10454Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 69m 0% 14344Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3747m 23% 3965Mi 6% 10:12:36 DEBUG --- stderr --- 10:12:36 DEBUG 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:13:26 DEBUG "pod": "ds-idrepo-0" 10:13:26 DEBUG }, 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:13:26 DEBUG "pod": "ds-idrepo-1" 10:13:26 DEBUG }, 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "ds-idrepo-0" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "ds-cts-1" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "ds-idrepo-0" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "ds-cts-1" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "ds-idrepo-0" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "ds-idrepo-1" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "ds-idrepo-0" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "ds-idrepo-1" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0.2" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "142389895.8518967" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "instance": "10.142.15.199:9100" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "instance": "10.142.15.212:9100" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "5516353536" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG { 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG "data": { 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "data": { 10:13:26 DEBUG "instance": "10.142.15.199:9100" 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG }, 10:13:26 DEBUG "result": [ 10:13:26 DEBUG "value": [ 10:13:26 DEBUG { 10:13:26 DEBUG 10:13:26 DEBUG "metric": { 10:13:26 DEBUG 10:13:26 DEBUG "pod": "ds-cts-2" 10:13:26 DEBUG 10:13:26 DEBUG }, 10:13:26 DEBUG [...] 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "instance": "10.142.15.212:9100" 10:13:26 DEBUG [...] 10:13:26 DEBUG }, 10:13:26 DEBUG 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG 10:13:26 DEBUG "26634.3" 10:13:26 DEBUG "pod": "ds-cts-1" 10:13:26 DEBUG ] 10:13:26 DEBUG }, 10:13:26 DEBUG } 10:13:26 DEBUG "value": [ 10:13:26 DEBUG ] 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG } 10:13:26 DEBUG "0" 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265206" 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:13:26 DEBUG "instance": "10.142.15.199:9100" 10:13:26 DEBUG }, 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:13:26 DEBUG "instance": "10.142.15.212:9100" 10:13:26 DEBUG }, 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG "28567.033333333333" 10:13:26 DEBUG ] 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265206" 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "instance": "10.142.15.199:9100" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "instance": "10.142.15.212:9100" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0.0019333333333333277" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "0" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "7.451842939208095" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265206" 10:13:26 INFO [http_cmd]: http status code OK 10:13:26 DEBUG --- status code --- 10:13:26 DEBUG http status code is 200 (expected 200) 10:13:26 DEBUG --- http response --- 10:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:13:26 DEBUG ----- output ----- 10:13:26 DEBUG { 10:13:26 DEBUG "status": "success", 10:13:26 DEBUG "data": { 10:13:26 DEBUG "resultType": "vector", 10:13:26 DEBUG "result": [ 10:13:26 DEBUG { 10:13:26 DEBUG "metric": { 10:13:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG [...] 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG 10:13:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:13:26 DEBUG }, 10:13:26 DEBUG "value": [ 10:13:26 DEBUG 1685265206, 10:13:26 DEBUG "4.535683688628556" 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG ] 10:13:26 DEBUG } 10:13:26 DEBUG } 10:13:34 INFO 10:13:34 INFO [loop_until]: kubectl --namespace=xlou top pods 10:13:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:13:34 INFO [loop_until]: OK (rc = 0) 10:13:34 DEBUG --- stdout --- 10:13:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 8m 5769Mi ds-cts-0 9m 460Mi ds-cts-1 9m 412Mi ds-cts-2 8m 427Mi ds-idrepo-0 7853m 9901Mi ds-idrepo-1 13m 13725Mi ds-idrepo-2 13m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11046m 5219Mi idm-58dc667486-h86fb 10342m 4911Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3660m 3043Mi 10:13:34 DEBUG --- stderr --- 10:13:34 DEBUG 10:13:36 INFO 10:13:36 INFO [loop_until]: kubectl --namespace=xlou top node 10:13:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:13:36 INFO [loop_until]: OK (rc = 0) 10:13:36 DEBUG --- stdout --- 10:13:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 95m 0% 6852Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 72m 0% 6657Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 118m 0% 1550Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6817Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15907m 100% 6606Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 10160m 63% 5993Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1353Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1095Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1107Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 57m 0% 1079Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13176Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7739m 48% 10451Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14342Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3750m 23% 3971Mi 6% 10:13:36 DEBUG --- stderr --- 10:13:36 DEBUG 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 DEBUG "pod": "ds-idrepo-0" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "ds-idrepo-1" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:14:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:14:26 DEBUG }, 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:14:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:14:26 DEBUG }, 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "ds-idrepo-0" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "ds-cts-1" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "ds-idrepo-0" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "ds-cts-1" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "ds-idrepo-0" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "ds-idrepo-1" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "ds-idrepo-0" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "ds-idrepo-1" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0.2" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "instance": "10.142.15.199:9100" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "instance": "10.142.15.212:9100" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0.0003333333333330302" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "138986647.30510727" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265266" 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265266" 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "instance": "10.142.15.199:9100" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "instance": "10.142.15.212:9100" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "28535.899999999998" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG { 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG { 10:14:26 DEBUG "data": { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "result": [ 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG { 10:14:26 DEBUG "result": [ 10:14:26 DEBUG "metric": { 10:14:26 DEBUG { 10:14:26 DEBUG "instance": "10.142.15.199:9100" 10:14:26 DEBUG "metric": { 10:14:26 DEBUG }, 10:14:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:14:26 DEBUG "value": [ 10:14:26 DEBUG }, 10:14:26 DEBUG 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "instance": "10.142.15.212:9100" 10:14:26 DEBUG 10:14:26 DEBUG }, 10:14:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:14:26 DEBUG "value": [ 10:14:26 DEBUG }, 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG "26837.266666666663" 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG ] 10:14:26 DEBUG "5534105600" 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "ds-cts-2" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "ds-cts-1" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265266" 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "instance": "10.142.15.199:9100" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "instance": "10.142.15.212:9100" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0.002466666666666839" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "0" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "7.088841094537009" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265266" 10:14:26 INFO [http_cmd]: http status code OK 10:14:26 DEBUG --- status code --- 10:14:26 DEBUG http status code is 200 (expected 200) 10:14:26 DEBUG --- http response --- 10:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:14:26 DEBUG ----- output ----- 10:14:26 DEBUG { 10:14:26 DEBUG "status": "success", 10:14:26 DEBUG "data": { 10:14:26 DEBUG "resultType": "vector", 10:14:26 DEBUG "result": [ 10:14:26 DEBUG { 10:14:26 DEBUG "metric": { 10:14:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG [...] 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG 10:14:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:14:26 DEBUG }, 10:14:26 DEBUG "value": [ 10:14:26 DEBUG 1685265266, 10:14:26 DEBUG "3.5972457155857436" 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG ] 10:14:26 DEBUG } 10:14:26 DEBUG } 10:14:34 INFO 10:14:34 INFO [loop_until]: kubectl --namespace=xlou top pods 10:14:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:14:34 INFO [loop_until]: OK (rc = 0) 10:14:34 DEBUG --- stdout --- 10:14:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 8m 5816Mi am-7849cf7bdb-vcl2f 6m 5769Mi ds-cts-0 8m 461Mi ds-cts-1 8m 412Mi ds-cts-2 8m 427Mi ds-idrepo-0 7806m 9901Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 14m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10983m 5219Mi idm-58dc667486-h86fb 10729m 4932Mi lodemon-cfd7686d9-6dlt4 3m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3540m 3038Mi 10:14:34 DEBUG --- stderr --- 10:14:34 DEBUG 10:14:37 INFO 10:14:37 INFO [loop_until]: kubectl --namespace=xlou top node 10:14:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:14:37 INFO [loop_until]: OK (rc = 0) 10:14:37 DEBUG --- stdout --- 10:14:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 99m 0% 6848Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 72m 0% 6659Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 132m 0% 1540Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 81m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15787m 99% 6391Mi 10% gke-xlou-cdm-default-pool-f05840a3-b0st 10687m 67% 6012Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 63m 0% 1096Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 59m 0% 1105Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1075Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 66m 0% 13179Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 8199m 51% 10452Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 71m 0% 14341Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3613m 22% 3967Mi 6% 10:14:37 DEBUG --- stderr --- 10:14:37 DEBUG 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "ds-idrepo-0" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "ds-idrepo-1" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG { 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:15:26 DEBUG "data": { 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG { 10:15:26 DEBUG "result": [ 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG { 10:15:26 DEBUG "data": { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:15:26 DEBUG "result": [ 10:15:26 DEBUG }, 10:15:26 DEBUG { 10:15:26 DEBUG "value": [ 10:15:26 DEBUG "metric": { 10:15:26 DEBUG 10:15:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:15:26 DEBUG 10:15:26 DEBUG }, 10:15:26 DEBUG 10:15:26 DEBUG "value": [ 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:15:26 DEBUG 10:15:26 DEBUG }, 10:15:26 DEBUG 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "pod": "ds-idrepo-2" 10:15:26 DEBUG "0" 10:15:26 DEBUG }, 10:15:26 DEBUG ] 10:15:26 DEBUG "value": [ 10:15:26 DEBUG } 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG ] 10:15:26 DEBUG "0.8883812424646235" 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "ds-idrepo-0" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "ds-cts-1" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "ds-idrepo-0" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "ds-idrepo-1" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "ds-idrepo-0" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "ds-cts-1" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "ds-idrepo-0" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "ds-idrepo-1" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0.2" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "instance": "10.142.15.199:9100" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "instance": "10.142.15.212:9100" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "ds-idrepo-2" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "15854.813122660069" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "ds-idrepo-2" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "instance": "10.142.15.199:9100" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "instance": "10.142.15.212:9100" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "28606.300000000003" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG { 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG "data": { 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "data": { 10:15:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG }, 10:15:26 DEBUG "result": [ 10:15:26 DEBUG "value": [ 10:15:26 DEBUG { 10:15:26 DEBUG 10:15:26 DEBUG "metric": { 10:15:26 DEBUG 10:15:26 DEBUG "instance": "10.142.15.199:9100" 10:15:26 DEBUG 10:15:26 DEBUG }, 10:15:26 DEBUG [...] 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:15:26 DEBUG [...] 10:15:26 DEBUG }, 10:15:26 DEBUG 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG 10:15:26 DEBUG "5563351040" 10:15:26 DEBUG "instance": "10.142.15.212:9100" 10:15:26 DEBUG ] 10:15:26 DEBUG }, 10:15:26 DEBUG } 10:15:26 DEBUG "value": [ 10:15:26 DEBUG ] 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG } 10:15:26 DEBUG "26691.5" 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "ds-cts-2" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "ds-cts-1" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG { 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG "data": { 10:15:26 DEBUG { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG "data": { 10:15:26 DEBUG { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "result": [ 10:15:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:15:26 DEBUG { 10:15:26 DEBUG }, 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "value": [ 10:15:26 DEBUG "instance": "10.142.15.199:9100" 10:15:26 DEBUG 10:15:26 DEBUG }, 10:15:26 DEBUG 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:15:26 DEBUG 10:15:26 DEBUG }, 10:15:26 DEBUG 10:15:26 DEBUG "value": [ 10:15:26 DEBUG "instance": "10.142.15.212:9100" 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG }, 10:15:26 DEBUG "0" 10:15:26 DEBUG "value": [ 10:15:26 DEBUG ] 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG } 10:15:26 DEBUG "0.0022000000000000834" 10:15:26 DEBUG ] 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265326" 10:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265326" 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "pod": "ds-idrepo-2" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "0.013230916714258259" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:26 INFO [http_cmd]: http status code OK 10:15:26 DEBUG --- status code --- 10:15:26 DEBUG http status code is 200 (expected 200) 10:15:26 DEBUG --- http response --- 10:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:15:26 DEBUG ----- output ----- 10:15:26 DEBUG { 10:15:26 DEBUG "status": "success", 10:15:26 DEBUG "data": { 10:15:26 DEBUG "resultType": "vector", 10:15:26 DEBUG "result": [ 10:15:26 DEBUG { 10:15:26 DEBUG "metric": { 10:15:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG [...] 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG 10:15:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:15:26 DEBUG }, 10:15:26 DEBUG "value": [ 10:15:26 DEBUG 1685265326, 10:15:26 DEBUG "3.5270395577767255" 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG ] 10:15:26 DEBUG } 10:15:26 DEBUG } 10:15:34 INFO 10:15:34 INFO [loop_until]: kubectl --namespace=xlou top pods 10:15:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:15:34 INFO [loop_until]: OK (rc = 0) 10:15:34 DEBUG --- stdout --- 10:15:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 6m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 8m 413Mi ds-cts-2 8m 427Mi ds-idrepo-0 7511m 9901Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 13m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11192m 5289Mi idm-58dc667486-h86fb 10259m 4943Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3605m 3041Mi 10:15:34 DEBUG --- stderr --- 10:15:34 DEBUG 10:15:37 INFO 10:15:37 INFO [loop_until]: kubectl --namespace=xlou top node 10:15:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:15:37 INFO [loop_until]: OK (rc = 0) 10:15:37 DEBUG --- stdout --- 10:15:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 95m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 67m 0% 6657Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 128m 0% 1543Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 79m 0% 6821Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15899m 100% 6484Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 9991m 62% 6031Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 78m 0% 1351Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1096Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1105Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 57m 0% 1080Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 64m 0% 13179Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7626m 47% 10452Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 68m 0% 14341Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3700m 23% 3968Mi 6% 10:15:37 DEBUG --- stderr --- 10:15:37 DEBUG 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "pod": "ds-idrepo-0" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "ds-cts-1" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG { 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG "data": { 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG "result": [ 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG { 10:16:26 DEBUG { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "data": { 10:16:26 DEBUG "pod": "ds-idrepo-0" 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG }, 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "pod": "ds-idrepo-0" 10:16:26 DEBUG "result": [ 10:16:26 DEBUG "value": [ 10:16:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:16:26 DEBUG }, 10:16:26 DEBUG { 10:16:26 DEBUG 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG "metric": { 10:16:26 DEBUG 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG }, 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "ds-idrepo-1" 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG }, 10:16:26 DEBUG 10:16:26 DEBUG "pod": "ds-idrepo-1" 10:16:26 DEBUG 10:16:26 DEBUG "value": [ 10:16:26 DEBUG "pod": "ds-idrepo-2" 10:16:26 DEBUG }, 10:16:26 DEBUG 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG "0" 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:16:26 DEBUG ] 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0" 10:16:26 DEBUG }, 10:16:26 DEBUG } 10:16:26 DEBUG "0.8534683186674121" 10:16:26 DEBUG ] 10:16:26 DEBUG "value": [ 10:16:26 DEBUG ] 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG "0" 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265386" 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "pod": "ds-idrepo-0" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "ds-idrepo-1" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "instance": "10.142.15.199:9100" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "instance": "10.142.15.212:9100" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG { 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG "data": { 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:16:26 DEBUG "result": [ 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:16:26 DEBUG "data": { 10:16:26 DEBUG }, 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "value": [ 10:16:26 DEBUG "result": [ 10:16:26 DEBUG 10:16:26 DEBUG { 10:16:26 DEBUG 10:16:26 DEBUG "metric": { 10:16:26 DEBUG 10:16:26 DEBUG "pod": "ds-idrepo-0" 10:16:26 DEBUG [...] 10:16:26 DEBUG }, 10:16:26 DEBUG 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:16:26 DEBUG 10:16:26 DEBUG }, 10:16:26 DEBUG [...] 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG 10:16:26 DEBUG "0.2" 10:16:26 DEBUG 10:16:26 DEBUG ] 10:16:26 DEBUG "pod": "ds-cts-1" 10:16:26 DEBUG } 10:16:26 DEBUG }, 10:16:26 DEBUG ] 10:16:26 DEBUG "value": [ 10:16:26 DEBUG } 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG } 10:16:26 DEBUG "0" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:16:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:16:26 DEBUG }, 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "ds-idrepo-2" 10:16:26 DEBUG "pod": "ds-idrepo-2" 10:16:26 DEBUG }, 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0" 10:16:26 DEBUG "19287.206804202593" 10:16:26 DEBUG ] 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "instance": "10.142.15.199:9100" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "instance": "10.142.15.212:9100" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "28573.766666666666" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265386" 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "5572583424" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "instance": "10.142.15.199:9100" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "instance": "10.142.15.212:9100" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "28273.533333333333" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "pod": "ds-cts-2" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "ds-cts-1" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265386" 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "3.574542904499504" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "instance": "10.142.15.199:9100" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "instance": "10.142.15.212:9100" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0.002066666666666824" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265386" 10:16:26 INFO [http_cmd]: http status code OK 10:16:26 DEBUG --- status code --- 10:16:26 DEBUG http status code is 200 (expected 200) 10:16:26 DEBUG --- http response --- 10:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:16:26 DEBUG ----- output ----- 10:16:26 DEBUG { 10:16:26 DEBUG "status": "success", 10:16:26 DEBUG "data": { 10:16:26 DEBUG "resultType": "vector", 10:16:26 DEBUG "result": [ 10:16:26 DEBUG { 10:16:26 DEBUG "metric": { 10:16:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG [...] 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG 10:16:26 DEBUG "pod": "ds-idrepo-2" 10:16:26 DEBUG }, 10:16:26 DEBUG "value": [ 10:16:26 DEBUG 1685265386, 10:16:26 DEBUG "0.014397868595301594" 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG ] 10:16:26 DEBUG } 10:16:26 DEBUG } 10:16:34 INFO 10:16:34 INFO [loop_until]: kubectl --namespace=xlou top pods 10:16:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:16:34 INFO [loop_until]: OK (rc = 0) 10:16:34 DEBUG --- stdout --- 10:16:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 6m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 9m 460Mi ds-cts-1 8m 413Mi ds-cts-2 8m 427Mi ds-idrepo-0 7234m 9901Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 15m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 10846m 5282Mi idm-58dc667486-h86fb 9524m 4952Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3372m 3044Mi 10:16:34 DEBUG --- stderr --- 10:16:34 DEBUG 10:16:37 INFO 10:16:37 INFO [loop_until]: kubectl --namespace=xlou top node 10:16:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:16:37 INFO [loop_until]: OK (rc = 0) 10:16:37 DEBUG --- stdout --- 10:16:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 93m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 68m 0% 6657Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 132m 0% 1548Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6824Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15924m 100% 6507Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 9891m 62% 6039Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 80m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1094Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1105Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 61m 0% 1081Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 69m 0% 13189Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7588m 47% 10454Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14342Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3473m 21% 3967Mi 6% 10:16:37 DEBUG --- stderr --- 10:16:37 DEBUG 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG { 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "result": [ 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG { 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "ds-idrepo-0" 10:17:26 DEBUG }, 10:17:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:17:26 DEBUG "pod": "ds-idrepo-0" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG }, 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG "value": [ 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "ds-idrepo-2" 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "ds-idrepo-1" 10:17:26 DEBUG }, 10:17:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:17:26 DEBUG "pod": "ds-cts-1" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG }, 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "value": [ 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0.8054913497233314" 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0" 10:17:26 DEBUG ] 10:17:26 DEBUG "0" 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG "0" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 DEBUG { 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG } 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "ds-idrepo-0" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "ds-idrepo-1" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "ds-idrepo-0" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG "pod": "ds-cts-1" 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG }, 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG "value": [ 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG "0" 10:17:26 DEBUG { 10:17:26 DEBUG ] 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG } 10:17:26 DEBUG "data": { 10:17:26 DEBUG ] 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG } 10:17:26 DEBUG "result": [ 10:17:26 DEBUG } 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "ds-idrepo-0" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "ds-idrepo-1" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "instance": "10.142.15.199:9100" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "instance": "10.142.15.212:9100" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265446" 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0.2" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "instance": "10.142.15.199:9100" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "instance": "10.142.15.212:9100" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "25789.6" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "5584355328" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:17:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:17:26 DEBUG }, 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "ds-idrepo-2" 10:17:26 DEBUG "pod": "ds-idrepo-2" 10:17:26 DEBUG }, 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0" 10:17:26 DEBUG "491820522.51873654" 10:17:26 DEBUG ] 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "instance": "10.142.15.199:9100" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "instance": "10.142.15.212:9100" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "28110.8" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "ds-cts-2" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "ds-cts-1" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "instance": "10.142.15.199:9100" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "instance": "10.142.15.212:9100" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0.0020333333333333314" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265446" 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "3.472958881521568" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265446" 10:17:26 INFO [http_cmd]: http status code OK 10:17:26 DEBUG --- status code --- 10:17:26 DEBUG http status code is 200 (expected 200) 10:17:26 DEBUG --- http response --- 10:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:17:26 DEBUG ----- output ----- 10:17:26 DEBUG { 10:17:26 DEBUG "status": "success", 10:17:26 DEBUG "data": { 10:17:26 DEBUG "resultType": "vector", 10:17:26 DEBUG "result": [ 10:17:26 DEBUG { 10:17:26 DEBUG "metric": { 10:17:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG [...] 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG 10:17:26 DEBUG "pod": "ds-idrepo-2" 10:17:26 DEBUG }, 10:17:26 DEBUG "value": [ 10:17:26 DEBUG 1685265446, 10:17:26 DEBUG "0.013187477446230179" 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG ] 10:17:26 DEBUG } 10:17:26 DEBUG } 10:17:34 INFO 10:17:34 INFO [loop_until]: kubectl --namespace=xlou top pods 10:17:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:17:34 INFO [loop_until]: OK (rc = 0) 10:17:34 DEBUG --- stdout --- 10:17:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 7m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 16m 460Mi ds-cts-1 8m 413Mi ds-cts-2 9m 427Mi ds-idrepo-0 7496m 9902Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 15m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 11285m 5375Mi idm-58dc667486-h86fb 9552m 4964Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 3416m 3048Mi 10:17:34 DEBUG --- stderr --- 10:17:34 DEBUG 10:17:37 INFO 10:17:37 INFO [loop_until]: kubectl --namespace=xlou top node 10:17:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:17:37 INFO [loop_until]: OK (rc = 0) 10:17:37 DEBUG --- stdout --- 10:17:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 91m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 71m 0% 6649Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 126m 0% 1543Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 15408m 96% 6581Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 9755m 61% 6050Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 81m 0% 1351Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1093Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1108Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 57m 0% 1083Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 67m 0% 13179Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 7542m 47% 10455Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14342Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 3543m 22% 3967Mi 6% 10:17:37 DEBUG --- stderr --- 10:17:37 DEBUG 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:18:26 DEBUG "pod": "ds-idrepo-0" 10:18:26 DEBUG }, 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG { 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG 10:18:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:18:26 DEBUG "data": { 10:18:26 DEBUG "pod": "ds-cts-1" 10:18:26 DEBUG }, 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG "result": [ 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG { 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG }, 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG "value": [ 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 DEBUG 10:18:26 DEBUG } 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "ds-idrepo-0" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "ds-idrepo-1" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "ds-idrepo-0" 10:18:26 DEBUG "pod": "ds-idrepo-0" 10:18:26 DEBUG }, 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "ds-idrepo-1" 10:18:26 DEBUG "pod": "ds-idrepo-1" 10:18:26 DEBUG }, 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "instance": "10.142.15.199:9100" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "instance": "10.142.15.212:9100" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:18:26 DEBUG "pod": "ds-idrepo-0" 10:18:26 DEBUG }, 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:18:26 DEBUG "pod": "ds-cts-1" 10:18:26 DEBUG }, 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0.2" 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "instance": "10.142.15.199:9100" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "instance": "10.142.15.212:9100" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "26375" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265506" 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "5595045888" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "instance": "10.142.15.199:9100" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "instance": "10.142.15.212:9100" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0.002733333333333595" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "instance": "10.142.15.199:9100" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "instance": "10.142.15.212:9100" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "28085.933333333334" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "ds-cts-2" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "ds-cts-1" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG { 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG "metric": { 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG }, 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:18:26 DEBUG "value": [ 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG 10:18:26 DEBUG { 10:18:26 DEBUG 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG 10:18:26 DEBUG "data": { 10:18:26 DEBUG [...] 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG 10:18:26 DEBUG "result": [ 10:18:26 DEBUG 10:18:26 DEBUG { 10:18:26 DEBUG 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:18:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:18:26 DEBUG }, 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG 10:18:26 DEBUG "0" 10:18:26 DEBUG 10:18:26 DEBUG ] 10:18:26 DEBUG 10:18:26 DEBUG } 10:18:26 DEBUG [...] 10:18:26 DEBUG ] 10:18:26 DEBUG 10:18:26 DEBUG } 10:18:26 DEBUG 10:18:26 DEBUG } 10:18:26 DEBUG 10:18:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "3.3573802677887676" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265506" 10:18:26 INFO [http_cmd]: http status code OK 10:18:26 DEBUG --- status code --- 10:18:26 DEBUG http status code is 200 (expected 200) 10:18:26 DEBUG --- http response --- 10:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:18:26 DEBUG ----- output ----- 10:18:26 DEBUG { 10:18:26 DEBUG "status": "success", 10:18:26 DEBUG "data": { 10:18:26 DEBUG "resultType": "vector", 10:18:26 DEBUG "result": [ 10:18:26 DEBUG { 10:18:26 DEBUG "metric": { 10:18:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG [...] 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG 10:18:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:18:26 DEBUG }, 10:18:26 DEBUG "value": [ 10:18:26 DEBUG 1685265506, 10:18:26 DEBUG "0" 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG ] 10:18:26 DEBUG } 10:18:26 DEBUG } 10:18:34 INFO 10:18:34 INFO [loop_until]: kubectl --namespace=xlou top pods 10:18:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:18:34 INFO [loop_until]: OK (rc = 0) 10:18:34 DEBUG --- stdout --- 10:18:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 6m 5769Mi ds-cts-0 9m 460Mi ds-cts-1 13m 414Mi ds-cts-2 14m 427Mi ds-idrepo-0 221m 9902Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 16m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 7m 5346Mi idm-58dc667486-h86fb 8m 4968Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 483m 3040Mi 10:18:34 DEBUG --- stderr --- 10:18:34 DEBUG 10:18:37 INFO 10:18:37 INFO [loop_until]: kubectl --namespace=xlou top node 10:18:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:18:37 INFO [loop_until]: OK (rc = 0) 10:18:37 DEBUG --- stdout --- 10:18:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 92m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 72m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 130m 0% 1545Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 90m 0% 6486Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 89m 0% 6065Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 80m 0% 1353Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 66m 0% 1095Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 59m 0% 1107Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1082Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 70m 0% 13179Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 82m 0% 10453Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14342Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 603m 3% 1300Mi 2% 10:18:37 DEBUG --- stderr --- 10:18:37 DEBUG 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG { 10:19:26 DEBUG { 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "data": { 10:19:26 DEBUG "data": { 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG "result": [ 10:19:26 DEBUG "result": [ 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG { 10:19:26 DEBUG { 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:19:26 DEBUG "pod": "ds-idrepo-0" 10:19:26 DEBUG "pod": "ds-idrepo-0" 10:19:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:19:26 DEBUG }, 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG }, 10:19:26 DEBUG }, 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG "value": [ 10:19:26 DEBUG "value": [ 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG [...] 10:19:26 DEBUG [...] 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG { 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "data": { 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "pod": "ds-idrepo-1" 10:19:26 DEBUG "pod": "ds-cts-1" 10:19:26 DEBUG "pod": "ds-idrepo-2" 10:19:26 DEBUG }, 10:19:26 DEBUG "result": [ 10:19:26 DEBUG }, 10:19:26 DEBUG }, 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG { 10:19:26 DEBUG "value": [ 10:19:26 DEBUG "value": [ 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "metric": { 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0" 10:19:26 DEBUG "pod": "ds-idrepo-0" 10:19:26 DEBUG "0" 10:19:26 DEBUG "0" 10:19:26 DEBUG "0.9507072685892081" 10:19:26 DEBUG ] 10:19:26 DEBUG }, 10:19:26 DEBUG ] 10:19:26 DEBUG ] 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG "value": [ 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG 10:19:26 DEBUG ] 10:19:26 DEBUG ] 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "pod": "ds-idrepo-1" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "instance": "10.142.15.199:9100" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "instance": "10.142.15.212:9100" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0.0003333333333330302" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "pod": "ds-idrepo-0" 10:19:26 DEBUG "pod": "ds-idrepo-0" 10:19:26 DEBUG }, 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "pod": "ds-idrepo-1" 10:19:26 DEBUG "pod": "ds-cts-1" 10:19:26 DEBUG }, 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0" 10:19:26 DEBUG "0" 10:19:26 DEBUG ] 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0.2" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265566" 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "instance": "10.142.15.199:9100" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265566" 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "instance": "10.142.15.212:9100" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "26430.666666666664" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "5590548480" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "pod": "ds-idrepo-2" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "pod": "ds-idrepo-2" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "404590421.24974793" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "instance": "10.142.15.199:9100" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "instance": "10.142.15.212:9100" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0.002666666666666373" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG { 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG "data": { 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "data": { 10:19:26 DEBUG "instance": "10.142.15.199:9100" 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG }, 10:19:26 DEBUG "result": [ 10:19:26 DEBUG "value": [ 10:19:26 DEBUG { 10:19:26 DEBUG 10:19:26 DEBUG "metric": { 10:19:26 DEBUG 10:19:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:19:26 DEBUG 10:19:26 DEBUG }, 10:19:26 DEBUG [...] 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG 10:19:26 DEBUG "instance": "10.142.15.212:9100" 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG [...] 10:19:26 DEBUG }, 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG 10:19:26 DEBUG "value": [ 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:19:26 DEBUG 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG 10:19:26 DEBUG "28254.8" 10:19:26 DEBUG { 10:19:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:19:26 DEBUG ] 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG }, 10:19:26 DEBUG } 10:19:26 DEBUG "data": { 10:19:26 DEBUG "value": [ 10:19:26 DEBUG ] 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG } 10:19:26 DEBUG "result": [ 10:19:26 DEBUG "0" 10:19:26 DEBUG } 10:19:26 DEBUG { 10:19:26 DEBUG ] 10:19:26 DEBUG "metric": { 10:19:26 DEBUG } 10:19:26 DEBUG "pod": "ds-cts-2" 10:19:26 DEBUG ] 10:19:26 DEBUG }, 10:19:26 DEBUG } 10:19:26 DEBUG "value": [ 10:19:26 DEBUG } 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "pod": "ds-cts-1" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265566" 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:19:26 DEBUG }, 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG "value": [ 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 DEBUG [...] 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG 10:19:26 DEBUG { 10:19:26 DEBUG 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG 10:19:26 DEBUG "data": { 10:19:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG }, 10:19:26 DEBUG "result": [ 10:19:26 DEBUG "value": [ 10:19:26 DEBUG { 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "0" 10:19:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:19:26 DEBUG ] 10:19:26 DEBUG }, 10:19:26 DEBUG } 10:19:26 DEBUG "value": [ 10:19:26 DEBUG ] 10:19:26 DEBUG 10:19:26 DEBUG } 10:19:26 DEBUG 10:19:26 DEBUG } 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0.205421552773782" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265566" 10:19:26 INFO [http_cmd]: http status code OK 10:19:26 DEBUG --- status code --- 10:19:26 DEBUG http status code is 200 (expected 200) 10:19:26 DEBUG --- http response --- 10:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:19:26 DEBUG ----- output ----- 10:19:26 DEBUG { 10:19:26 DEBUG "status": "success", 10:19:26 DEBUG "data": { 10:19:26 DEBUG "resultType": "vector", 10:19:26 DEBUG "result": [ 10:19:26 DEBUG { 10:19:26 DEBUG "metric": { 10:19:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG [...] 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG 10:19:26 DEBUG "pod": "ds-idrepo-2" 10:19:26 DEBUG }, 10:19:26 DEBUG "value": [ 10:19:26 DEBUG 1685265566, 10:19:26 DEBUG "0.017497679870937127" 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG ] 10:19:26 DEBUG } 10:19:26 DEBUG } 10:19:34 INFO 10:19:34 INFO [loop_until]: kubectl --namespace=xlou top pods 10:19:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:19:34 INFO [loop_until]: OK (rc = 0) 10:19:34 DEBUG --- stdout --- 10:19:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 9m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 10m 413Mi ds-cts-2 8m 428Mi ds-idrepo-0 20m 9902Mi ds-idrepo-1 14m 13725Mi ds-idrepo-2 17m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 7m 5341Mi idm-58dc667486-h86fb 7m 4967Mi lodemon-cfd7686d9-6dlt4 2m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 1m 358Mi 10:19:34 DEBUG --- stderr --- 10:19:34 DEBUG 10:19:37 INFO 10:19:37 INFO [loop_until]: kubectl --namespace=xlou top node 10:19:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:19:37 INFO [loop_until]: OK (rc = 0) 10:19:37 DEBUG --- stdout --- 10:19:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 92m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 69m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 126m 0% 1550Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 82m 0% 6486Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 84m 0% 6056Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 83m 0% 1348Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1097Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1103Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 60m 0% 1083Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 70m 0% 13181Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 81m 0% 10454Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14341Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 70m 0% 1300Mi 2% 10:19:37 DEBUG --- stderr --- 10:19:37 DEBUG 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265626" 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "pod": "ds-idrepo-0" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "ds-idrepo-1" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG { 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG "data": { 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "data": { 10:20:26 DEBUG "pod": "ds-idrepo-0" 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG }, 10:20:26 DEBUG "result": [ 10:20:26 DEBUG "value": [ 10:20:26 DEBUG { 10:20:26 DEBUG 10:20:26 DEBUG "metric": { 10:20:26 DEBUG 10:20:26 DEBUG "instance": "10.142.15.199:9100" 10:20:26 DEBUG 10:20:26 DEBUG }, 10:20:26 DEBUG [...] 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "ds-idrepo-1" 10:20:26 DEBUG [...] 10:20:26 DEBUG }, 10:20:26 DEBUG 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG 10:20:26 DEBUG "0" 10:20:26 DEBUG "instance": "10.142.15.212:9100" 10:20:26 DEBUG ] 10:20:26 DEBUG }, 10:20:26 DEBUG } 10:20:26 DEBUG "value": [ 10:20:26 DEBUG ] 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG } 10:20:26 DEBUG "0" 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG { 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:20:26 DEBUG "data": { 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG { 10:20:26 DEBUG "result": [ 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG { 10:20:26 DEBUG "data": { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "pod": "ds-idrepo-0" 10:20:26 DEBUG "result": [ 10:20:26 DEBUG }, 10:20:26 DEBUG { 10:20:26 DEBUG "value": [ 10:20:26 DEBUG "metric": { 10:20:26 DEBUG 10:20:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:20:26 DEBUG 10:20:26 DEBUG }, 10:20:26 DEBUG 10:20:26 DEBUG "value": [ 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG "pod": "ds-cts-1" 10:20:26 DEBUG 10:20:26 DEBUG }, 10:20:26 DEBUG 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "pod": "ds-cts-0" 10:20:26 DEBUG "0" 10:20:26 DEBUG }, 10:20:26 DEBUG ] 10:20:26 DEBUG "value": [ 10:20:26 DEBUG } 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG ] 10:20:26 DEBUG "0" 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "pod": "ds-idrepo-0" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "ds-idrepo-1" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:20:26 DEBUG "pod": "ds-idrepo-0" 10:20:26 DEBUG }, 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:20:26 DEBUG "pod": "ds-cts-1" 10:20:26 DEBUG }, 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0.2" 10:20:26 DEBUG "0" 10:20:26 DEBUG ] 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265626" 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "instance": "10.142.15.199:9100" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "instance": "10.142.15.212:9100" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "27201.333333333332" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265626" 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "5591109632" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:20:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:20:26 DEBUG }, 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:20:26 DEBUG "pod": "ds-cts-0" 10:20:26 DEBUG }, 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0" 10:20:26 DEBUG "0" 10:20:26 DEBUG ] 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "ds-cts-0" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "4860.635656554674" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265626" 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265626" 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "instance": "10.142.15.199:9100" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "instance": "10.142.15.212:9100" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0.002799999999999869" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "pod": "ds-cts-2" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "ds-cts-1" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "instance": "10.142.15.199:9100" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "instance": "10.142.15.212:9100" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "41012.7" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265626" 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG "metric": { 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG }, 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG "value": [ 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:20:26 DEBUG 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG 10:20:26 DEBUG { 10:20:26 DEBUG 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG [...] 10:20:26 DEBUG "data": { 10:20:26 DEBUG 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG 10:20:26 DEBUG "result": [ 10:20:26 DEBUG 10:20:26 DEBUG { 10:20:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:20:26 DEBUG "metric": { 10:20:26 DEBUG }, 10:20:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:20:26 DEBUG "value": [ 10:20:26 DEBUG }, 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG "0" 10:20:26 DEBUG 10:20:26 DEBUG ] 10:20:26 DEBUG 10:20:26 DEBUG } 10:20:26 DEBUG 10:20:26 DEBUG ] 10:20:26 DEBUG [...] 10:20:26 DEBUG } 10:20:26 DEBUG 10:20:26 DEBUG } 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0.027787753628486477" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265626" 10:20:26 INFO [http_cmd]: http status code OK 10:20:26 DEBUG --- status code --- 10:20:26 DEBUG http status code is 200 (expected 200) 10:20:26 DEBUG --- http response --- 10:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:20:26 DEBUG ----- output ----- 10:20:26 DEBUG { 10:20:26 DEBUG "status": "success", 10:20:26 DEBUG "data": { 10:20:26 DEBUG "resultType": "vector", 10:20:26 DEBUG "result": [ 10:20:26 DEBUG { 10:20:26 DEBUG "metric": { 10:20:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG [...] 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG 10:20:26 DEBUG "pod": "ds-cts-0" 10:20:26 DEBUG }, 10:20:26 DEBUG "value": [ 10:20:26 DEBUG 1685265626, 10:20:26 DEBUG "0.003860460730958102" 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG ] 10:20:26 DEBUG } 10:20:26 DEBUG } 10:20:35 INFO 10:20:35 INFO [loop_until]: kubectl --namespace=xlou top pods 10:20:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:20:35 INFO [loop_until]: OK (rc = 0) 10:20:35 DEBUG --- stdout --- 10:20:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 4Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 6m 5769Mi ds-cts-0 8m 461Mi ds-cts-1 8m 414Mi ds-cts-2 8m 428Mi ds-idrepo-0 20m 9902Mi ds-idrepo-1 15m 13725Mi ds-idrepo-2 15m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 3Mi idm-58dc667486-gclr8 6m 5341Mi idm-58dc667486-h86fb 6m 4967Mi lodemon-cfd7686d9-6dlt4 1m 66Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 2m 358Mi 10:20:35 DEBUG --- stderr --- 10:20:35 DEBUG 10:20:37 INFO 10:20:37 INFO [loop_until]: kubectl --namespace=xlou top node 10:20:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:20:37 INFO [loop_until]: OK (rc = 0) 10:20:37 DEBUG --- stdout --- 10:20:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 96m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 73m 0% 6656Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 114m 0% 1552Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 87m 0% 6821Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 82m 0% 6483Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 88m 0% 6058Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 89m 0% 1349Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 62m 0% 1092Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 55m 0% 1104Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1081Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 70m 0% 13181Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 78m 0% 10454Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 68m 0% 14343Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 77m 0% 1302Mi 2% 10:20:37 DEBUG --- stderr --- 10:20:37 DEBUG 127.0.0.1 - - [28/May/2023 10:21:05] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:21:11] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:21:18] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:21:24] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "ds-idrepo-0" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "ds-idrepo-1" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "ds-idrepo-0" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "ds-idrepo-1" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0.36945916663724254" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265686" 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "instance": "10.142.15.199:9100" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "instance": "10.142.15.212:9100" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0.0003333333333330302" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "ds-idrepo-0" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "ds-cts-1" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "ds-idrepo-0" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG "pod": "ds-cts-1" 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG }, 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG "value": [ 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG "0" 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG ] 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG } 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG ] 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG } 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG } 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "ds-idrepo-0" 10:21:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:21:26 DEBUG }, 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "ds-idrepo-1" 10:21:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:21:26 DEBUG }, 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG "0.2" 10:21:26 DEBUG ] 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265686" 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "5590122496" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "instance": "10.142.15.199:9100" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "instance": "10.142.15.212:9100" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "28752.433333333334" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "82695168.18415464" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265686" 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "instance": "10.142.15.199:9100" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "instance": "10.142.15.212:9100" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "28510.233333333334" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "instance": "10.142.15.199:9100" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "instance": "10.142.15.212:9100" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0.0032666666666668694" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "ds-cts-2" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "ds-cts-1" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "0" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "1.2328694926423711" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 10:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265686" 10:21:26 INFO [http_cmd]: http status code OK 10:21:26 DEBUG --- status code --- 10:21:26 DEBUG http status code is 200 (expected 200) 10:21:26 DEBUG --- http response --- 10:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:21:26 DEBUG ----- output ----- 10:21:26 DEBUG { 10:21:26 DEBUG "status": "success", 10:21:26 DEBUG "data": { 10:21:26 DEBUG "resultType": "vector", 10:21:26 DEBUG "result": [ 10:21:26 DEBUG { 10:21:26 DEBUG "metric": { 10:21:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG [...] 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG 10:21:26 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:21:26 DEBUG }, 10:21:26 DEBUG "value": [ 10:21:26 DEBUG 1685265686, 10:21:26 DEBUG "1.0470707563672137" 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG ] 10:21:26 DEBUG } 10:21:26 DEBUG } 127.0.0.1 - - [28/May/2023 10:21:31] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 10:21:35 INFO 10:21:35 INFO [loop_until]: kubectl --namespace=xlou top pods 10:21:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:21:35 INFO [loop_until]: OK (rc = 0) 10:21:35 DEBUG --- stdout --- 10:21:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 6Mi am-7849cf7bdb-72wqt 10m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 8m 414Mi ds-cts-2 9m 428Mi ds-idrepo-0 316m 9979Mi ds-idrepo-1 162m 13713Mi ds-idrepo-2 173m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 5Mi idm-58dc667486-gclr8 7m 5341Mi idm-58dc667486-h86fb 13m 4967Mi lodemon-cfd7686d9-6dlt4 7m 68Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 801m 785Mi 10:21:35 DEBUG --- stderr --- 10:21:35 DEBUG 10:21:37 INFO 10:21:37 INFO [loop_until]: kubectl --namespace=xlou top node 10:21:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 127.0.0.1 - - [28/May/2023 10:21:37] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 10:21:37 INFO [loop_until]: OK (rc = 0) 10:21:37 DEBUG --- stdout --- 10:21:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 96m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 71m 0% 6653Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 133m 0% 1556Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 73m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 84m 0% 6486Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 81m 0% 6051Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 95m 0% 1350Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1096Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1104Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1080Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 208m 1% 13182Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 335m 2% 10458Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 190m 1% 14334Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 809m 5% 1711Mi 2% 10:21:37 DEBUG --- stderr --- 10:21:37 DEBUG 127.0.0.1 - - [28/May/2023 10:21:44] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:21:51] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:21:57] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:21:58] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:04] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:05] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:12] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:12] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:19] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:20] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "ds-idrepo-0" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "ds-idrepo-1" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265746" 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 127.0.0.1 - - [28/May/2023 10:22:26] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG { 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:22:26 DEBUG "data": { 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG { 10:22:26 DEBUG { 10:22:26 DEBUG "result": [ 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG { 10:22:26 DEBUG "data": { 10:22:26 DEBUG "data": { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "pod": "ds-idrepo-0" 10:22:26 DEBUG "result": [ 10:22:26 DEBUG "result": [ 10:22:26 DEBUG }, 10:22:26 DEBUG { 10:22:26 DEBUG { 10:22:26 DEBUG "value": [ 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG 10:22:26 DEBUG "pod": "ds-idrepo-0" 10:22:26 DEBUG "instance": "10.142.15.199:9100" 10:22:26 DEBUG 10:22:26 DEBUG }, 10:22:26 DEBUG }, 10:22:26 DEBUG 10:22:26 DEBUG "value": [ 10:22:26 DEBUG "value": [ 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG [...] 10:22:26 DEBUG "pod": "ds-idrepo-1" 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG }, 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "pod": "ds-idrepo-1" 10:22:26 DEBUG "instance": "10.142.15.212:9100" 10:22:26 DEBUG "0" 10:22:26 DEBUG }, 10:22:26 DEBUG }, 10:22:26 DEBUG ] 10:22:26 DEBUG "value": [ 10:22:26 DEBUG "value": [ 10:22:26 DEBUG } 10:22:26 DEBUG 1685265746, 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG ] 10:22:26 DEBUG "0" 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG "0.0006666666666660604" 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 DEBUG { 10:22:26 DEBUG } 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "ds-idrepo-0" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "ds-cts-1" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "ds-idrepo-0" 10:22:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:22:26 DEBUG }, 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "ds-cts-1" 10:22:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:22:26 DEBUG }, 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG "0.2" 10:22:26 DEBUG ] 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265746" 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "5589811200" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265746" 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "instance": "10.142.15.199:9100" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "instance": "10.142.15.212:9100" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "26352.433333333334" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "189797669.58695096" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265746" 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265746" 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "instance": "10.142.15.199:9100" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "instance": "10.142.15.212:9100" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0.0029000000000003467" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "instance": "10.142.15.199:9100" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "instance": "10.142.15.212:9100" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "28090.933333333334" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "ds-cts-2" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "ds-cts-1" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265746" 10:22:26 INFO [http_cmd]: http status code OK 10:22:26 DEBUG --- status code --- 10:22:26 DEBUG http status code is 200 (expected 200) 10:22:26 DEBUG --- http response --- 10:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:22:26 DEBUG ----- output ----- 10:22:26 DEBUG { 10:22:26 DEBUG "status": "success", 10:22:26 DEBUG "data": { 10:22:26 DEBUG "resultType": "vector", 10:22:26 DEBUG "result": [ 10:22:26 DEBUG { 10:22:26 DEBUG "metric": { 10:22:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG [...] 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG 10:22:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:22:26 DEBUG }, 10:22:26 DEBUG "value": [ 10:22:26 DEBUG 1685265746, 10:22:26 DEBUG "0" 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG ] 10:22:26 DEBUG } 10:22:26 DEBUG } 10:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265746" 10:22:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265747" 10:22:27 INFO [http_cmd]: http status code OK 10:22:27 DEBUG --- status code --- 10:22:27 DEBUG http status code is 200 (expected 200) 10:22:27 DEBUG --- http response --- 10:22:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:22:27 DEBUG ----- output ----- 10:22:27 DEBUG { 10:22:27 DEBUG "status": "success", 10:22:27 DEBUG "data": { 10:22:27 DEBUG "resultType": "vector", 10:22:27 DEBUG "result": [ 10:22:27 DEBUG { 10:22:27 DEBUG "metric": { 10:22:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:22:27 DEBUG }, 10:22:27 DEBUG "value": [ 10:22:27 DEBUG 10:22:27 DEBUG 10:22:27 DEBUG 10:22:27 DEBUG [...] 10:22:27 DEBUG 10:22:27 DEBUG 10:22:27 DEBUG 10:22:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:22:27 DEBUG }, 10:22:27 DEBUG "value": [ 10:22:27 DEBUG 1685265746, 10:22:27 DEBUG "1.0990462719015937" 10:22:27 DEBUG ] 10:22:27 DEBUG } 10:22:27 DEBUG ] 10:22:27 DEBUG } 10:22:27 DEBUG } 10:22:27 INFO [http_cmd]: http status code OK 10:22:27 DEBUG --- status code --- 10:22:27 DEBUG http status code is 200 (expected 200) 10:22:27 DEBUG --- http response --- 10:22:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:22:27 DEBUG ----- output ----- 10:22:27 DEBUG { 10:22:27 DEBUG "status": "success", 10:22:27 DEBUG "data": { 10:22:27 DEBUG "resultType": "vector", 10:22:27 DEBUG "result": [ 10:22:27 DEBUG { 10:22:27 DEBUG "metric": { 10:22:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:22:27 DEBUG }, 10:22:27 DEBUG "value": [ 10:22:27 DEBUG 10:22:27 DEBUG 10:22:27 DEBUG 10:22:27 DEBUG [...] 10:22:27 DEBUG 10:22:27 DEBUG 10:22:27 DEBUG 10:22:27 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:22:27 DEBUG }, 10:22:27 DEBUG "value": [ 10:22:27 DEBUG 1685265747, 10:22:27 DEBUG "0.004379263305812915" 10:22:27 DEBUG ] 10:22:27 DEBUG } 10:22:27 DEBUG ] 10:22:27 DEBUG } 10:22:27 DEBUG } 127.0.0.1 - - [28/May/2023 10:22:27] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:33] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:34] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 10:22:35 INFO 10:22:35 INFO [loop_until]: kubectl --namespace=xlou top pods 10:22:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:22:35 INFO [loop_until]: OK (rc = 0) 10:22:35 DEBUG --- stdout --- 10:22:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 6Mi am-7849cf7bdb-72wqt 9m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 8m 414Mi ds-cts-2 9m 428Mi ds-idrepo-0 31m 9903Mi ds-idrepo-1 18m 13713Mi ds-idrepo-2 15m 12554Mi end-user-ui-fdf99b7d7-4gw9n 1m 5Mi idm-58dc667486-gclr8 7m 5341Mi idm-58dc667486-h86fb 7m 4967Mi lodemon-cfd7686d9-6dlt4 17m 68Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 806m 943Mi 10:22:35 DEBUG --- stderr --- 10:22:35 DEBUG 10:22:37 INFO 10:22:37 INFO [loop_until]: kubectl --namespace=xlou top node 10:22:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:22:38 INFO [loop_until]: OK (rc = 0) 10:22:38 DEBUG --- stdout --- 10:22:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 93m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 68m 0% 6656Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 128m 0% 1545Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 74m 0% 6821Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 89m 0% 6485Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 82m 0% 6056Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 106m 0% 1353Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1095Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 56m 0% 1106Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 62m 0% 1090Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 68m 0% 13181Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 90m 0% 10459Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 69m 0% 14335Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 1149m 7% 1883Mi 3% 10:22:38 DEBUG --- stderr --- 10:22:38 DEBUG 127.0.0.1 - - [28/May/2023 10:22:41] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:42] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:48] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:49] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:56] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:22:56] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:03] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:04] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:05] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:11] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:12] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:13] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:19] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:20] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:21] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "ds-idrepo-0" 10:23:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:23:26 DEBUG }, 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "ds-idrepo-1" 10:23:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:23:26 DEBUG }, 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265806" 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "instance": "10.142.15.199:9100" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "instance": "10.142.15.212:9100" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0.0003333333333330302" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "ds-idrepo-0" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "ds-cts-1" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265806" 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "ds-idrepo-0" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "ds-idrepo-1" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:23:26 DEBUG "pod": "ds-idrepo-0" 10:23:26 DEBUG }, 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:23:26 DEBUG "pod": "ds-idrepo-1" 10:23:26 DEBUG }, 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "5591818240" 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "ds-idrepo-0" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "ds-cts-1" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0.2" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "instance": "10.142.15.199:9100" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "instance": "10.142.15.212:9100" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "26288.6" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265806" 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG "data": { 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG "result": [ 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG { 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:23:26 DEBUG "metric": { 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:23:26 DEBUG { 10:23:26 DEBUG }, 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "value": [ 10:23:26 DEBUG "data": { 10:23:26 DEBUG 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG 10:23:26 DEBUG "result": [ 10:23:26 DEBUG 10:23:26 DEBUG { 10:23:26 DEBUG [...] 10:23:26 DEBUG "metric": { 10:23:26 DEBUG 10:23:26 DEBUG "instance": "10.142.15.199:9100" 10:23:26 DEBUG 10:23:26 DEBUG }, 10:23:26 DEBUG 10:23:26 DEBUG "value": [ 10:23:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:23:26 DEBUG 10:23:26 DEBUG }, 10:23:26 DEBUG 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG [...] 10:23:26 DEBUG "0" 10:23:26 DEBUG 10:23:26 DEBUG ] 10:23:26 DEBUG 10:23:26 DEBUG } 10:23:26 DEBUG 10:23:26 DEBUG ] 10:23:26 DEBUG "instance": "10.142.15.212:9100" 10:23:26 DEBUG } 10:23:26 DEBUG }, 10:23:26 DEBUG } 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0.0027666666666666135" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "instance": "10.142.15.199:9100" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "instance": "10.142.15.212:9100" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "28023.666666666668" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "ds-cts-2" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "ds-cts-1" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265806" 10:23:26 INFO [http_cmd]: http status code OK 10:23:26 DEBUG --- status code --- 10:23:26 DEBUG http status code is 200 (expected 200) 10:23:26 DEBUG --- http response --- 10:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:26 DEBUG ----- output ----- 10:23:26 DEBUG { 10:23:26 DEBUG "status": "success", 10:23:26 DEBUG "data": { 10:23:26 DEBUG "resultType": "vector", 10:23:26 DEBUG "result": [ 10:23:26 DEBUG { 10:23:26 DEBUG "metric": { 10:23:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG [...] 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG 10:23:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:23:26 DEBUG }, 10:23:26 DEBUG "value": [ 10:23:26 DEBUG 1685265806, 10:23:26 DEBUG "0" 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG ] 10:23:26 DEBUG } 10:23:26 DEBUG } 10:23:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265807" 10:23:27 INFO [http_cmd]: http status code OK 10:23:27 DEBUG --- status code --- 10:23:27 DEBUG http status code is 200 (expected 200) 10:23:27 DEBUG --- http response --- 10:23:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:23:27 DEBUG ----- output ----- 10:23:27 DEBUG { 10:23:27 DEBUG "status": "success", 10:23:27 DEBUG "data": { 10:23:27 DEBUG "resultType": "vector", 10:23:27 DEBUG "result": [ 10:23:27 DEBUG { 10:23:27 DEBUG "metric": { 10:23:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:23:27 DEBUG }, 10:23:27 DEBUG "value": [ 10:23:27 DEBUG 10:23:27 DEBUG 10:23:27 DEBUG 10:23:27 DEBUG [...] 10:23:27 DEBUG 10:23:27 DEBUG 10:23:27 DEBUG 10:23:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:23:27 DEBUG }, 10:23:27 DEBUG "value": [ 10:23:27 DEBUG 1685265807, 10:23:27 DEBUG "0.7873391939138995" 10:23:27 DEBUG ] 10:23:27 DEBUG } 10:23:27 DEBUG ] 10:23:27 DEBUG } 10:23:27 DEBUG } 10:23:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265807" 10:23:27 INFO [http_cmd]: http status code OK 10:23:27 DEBUG --- status code --- 10:23:27 DEBUG http status code is 200 (expected 200) 10:23:27 DEBUG --- http response --- 10:23:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:23:27 DEBUG ----- output ----- 10:23:27 DEBUG { 10:23:27 DEBUG "status": "success", 10:23:27 DEBUG "data": { 10:23:27 DEBUG "resultType": "vector", 10:23:27 DEBUG "result": [ 10:23:27 DEBUG { 10:23:27 DEBUG "metric": { 10:23:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:23:27 DEBUG }, 10:23:27 DEBUG "value": [ 10:23:27 DEBUG 10:23:27 DEBUG 10:23:27 DEBUG 10:23:27 DEBUG [...] 10:23:27 DEBUG 10:23:27 DEBUG 10:23:27 DEBUG 10:23:27 DEBUG "pod": "am-7849cf7bdb-72wqt" 10:23:27 DEBUG }, 10:23:27 DEBUG "value": [ 10:23:27 DEBUG 1685265807, 10:23:27 DEBUG "0.00794874462229286" 10:23:27 DEBUG ] 10:23:27 DEBUG } 10:23:27 DEBUG ] 10:23:27 DEBUG } 10:23:27 DEBUG } 127.0.0.1 - - [28/May/2023 10:23:27] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:28] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:29] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 10:23:35 INFO 10:23:35 INFO [loop_until]: kubectl --namespace=xlou top pods 10:23:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:23:35 INFO [loop_until]: OK (rc = 0) 10:23:35 DEBUG --- stdout --- 10:23:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 6Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 9m 414Mi ds-cts-2 8m 428Mi ds-idrepo-0 26m 9904Mi ds-idrepo-1 17m 13713Mi ds-idrepo-2 15m 12551Mi end-user-ui-fdf99b7d7-4gw9n 1m 5Mi idm-58dc667486-gclr8 7m 5341Mi idm-58dc667486-h86fb 6m 4967Mi lodemon-cfd7686d9-6dlt4 26m 68Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 780m 1006Mi 10:23:35 DEBUG --- stderr --- 10:23:35 DEBUG 127.0.0.1 - - [28/May/2023 10:23:35] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:36] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:37] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 10:23:38 INFO 10:23:38 INFO [loop_until]: kubectl --namespace=xlou top node 10:23:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:23:38 INFO [loop_until]: OK (rc = 0) 10:23:38 DEBUG --- stdout --- 10:23:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 90m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6657Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 118m 0% 1544Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 76m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 85m 0% 6482Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 80m 0% 6057Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 116m 0% 1356Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 60m 0% 1097Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1103Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1078Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 68m 0% 13176Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 82m 0% 10459Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 68m 0% 14334Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 947m 5% 1989Mi 3% 10:23:38 DEBUG --- stderr --- 10:23:38 DEBUG 127.0.0.1 - - [28/May/2023 10:23:44] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:44] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:45] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:52] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:52] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:23:53] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:00] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:01] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:01] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:08] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:09] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:10] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:16] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:17] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:18] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:18] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:25] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:26] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "pod": "ds-idrepo-0" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "pod": "ds-idrepo-1" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "0" 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "pod": "ds-idrepo-2" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "0.9409274015701726" 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265866" 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG { 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG "data": { 10:24:26 DEBUG { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG "data": { 10:24:26 DEBUG { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "result": [ 10:24:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:24:26 DEBUG { 10:24:26 DEBUG }, 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "value": [ 10:24:26 DEBUG "instance": "10.142.15.199:9100" 10:24:26 DEBUG 10:24:26 DEBUG }, 10:24:26 DEBUG 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:24:26 DEBUG 10:24:26 DEBUG }, 10:24:26 DEBUG 10:24:26 DEBUG "value": [ 10:24:26 DEBUG "instance": "10.142.15.212:9100" 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG }, 10:24:26 DEBUG "0" 10:24:26 DEBUG "value": [ 10:24:26 DEBUG ] 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG } 10:24:26 DEBUG "0" 10:24:26 DEBUG ] 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "pod": "ds-idrepo-0" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "pod": "ds-cts-1" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "0" 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265866" 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265866" 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "5593358336" 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 DEBUG { 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG { 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG "result": [ 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG { 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "pod": "ds-idrepo-0" 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "pod": "ds-idrepo-0" 10:24:26 DEBUG }, 10:24:26 DEBUG "instance": "10.142.15.199:9100" 10:24:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG }, 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG "value": [ 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [...] 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "pod": "ds-idrepo-1" 10:24:26 DEBUG { 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "pod": "ds-idrepo-1" 10:24:26 DEBUG }, 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "instance": "10.142.15.212:9100" 10:24:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG "data": { 10:24:26 DEBUG }, 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "value": [ 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "0" 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG "result": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "0" 10:24:26 DEBUG ] 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG { 10:24:26 DEBUG "26342" 10:24:26 DEBUG "0.2" 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG "metric": { 10:24:26 DEBUG ] 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:24:26 DEBUG }, 10:24:26 DEBUG ] 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG "value": [ 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG { 10:24:26 DEBUG 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG 10:24:26 DEBUG "data": { 10:24:26 DEBUG 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG [...] 10:24:26 DEBUG "result": [ 10:24:26 DEBUG 10:24:26 DEBUG { 10:24:26 DEBUG 10:24:26 DEBUG "metric": { 10:24:26 DEBUG 10:24:26 DEBUG "pod": "ds-idrepo-0" 10:24:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:24:26 DEBUG }, 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG 10:24:26 DEBUG "0" 10:24:26 DEBUG 10:24:26 DEBUG ] 10:24:26 DEBUG [...] 10:24:26 DEBUG } 10:24:26 DEBUG 10:24:26 DEBUG ] 10:24:26 DEBUG 10:24:26 DEBUG } 10:24:26 DEBUG 10:24:26 DEBUG } 10:24:26 DEBUG "pod": "ds-cts-1" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "0" 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "pod": "ds-idrepo-2" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "0" 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265866" 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265866" 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "pod": "ds-idrepo-2" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "20354.141550765973" 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG { 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "instance": "10.142.15.199:9100" 10:24:26 DEBUG "instance": "10.142.15.199:9100" 10:24:26 DEBUG }, 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG "instance": "10.142.15.212:9100" 10:24:26 DEBUG "instance": "10.142.15.212:9100" 10:24:26 DEBUG }, 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG 1685265866, 10:24:26 DEBUG "0.002700000000000102" 10:24:26 DEBUG "28060.566666666666" 10:24:26 DEBUG ] 10:24:26 DEBUG ] 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG ] 10:24:26 DEBUG ] 10:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265866" 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 DEBUG } 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG --- status code --- 10:24:26 DEBUG http status code is 200 (expected 200) 10:24:26 DEBUG --- http response --- 10:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:26 DEBUG ----- output ----- 10:24:26 DEBUG { 10:24:26 DEBUG "status": "success", 10:24:26 DEBUG "data": { 10:24:26 DEBUG "resultType": "vector", 10:24:26 DEBUG "result": [ 10:24:26 DEBUG { 10:24:26 DEBUG "metric": { 10:24:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:24:26 DEBUG }, 10:24:26 DEBUG "value": [ 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG [...] 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 DEBUG 10:24:26 INFO [http_cmd]: http status code OK 10:24:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:24:27 DEBUG --- status code --- 10:24:27 DEBUG }, 10:24:27 DEBUG http status code is 200 (expected 200) 10:24:27 DEBUG "value": [ 10:24:27 DEBUG --- http response --- 10:24:27 DEBUG 1685265866, 10:24:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:27 DEBUG "0" 10:24:27 DEBUG ----- output ----- 10:24:27 DEBUG ] 10:24:27 DEBUG { 10:24:27 DEBUG } 10:24:27 DEBUG "status": "success", 10:24:27 DEBUG ] 10:24:27 DEBUG "data": { 10:24:27 DEBUG } 10:24:27 DEBUG "resultType": "vector", 10:24:27 DEBUG } 10:24:27 DEBUG "result": [ 10:24:27 DEBUG { 10:24:27 DEBUG "metric": { 10:24:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:24:27 DEBUG }, 10:24:27 DEBUG "value": [ 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG [...] 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:24:27 DEBUG }, 10:24:27 DEBUG "value": [ 10:24:27 DEBUG 1685265866, 10:24:27 DEBUG "0" 10:24:27 DEBUG ] 10:24:27 DEBUG } 10:24:27 DEBUG ] 10:24:27 DEBUG } 10:24:27 DEBUG } 10:24:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265867" 10:24:27 INFO [http_cmd]: http status code OK 10:24:27 DEBUG --- status code --- 10:24:27 DEBUG http status code is 200 (expected 200) 10:24:27 DEBUG --- http response --- 10:24:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:24:27 DEBUG ----- output ----- 10:24:27 DEBUG { 10:24:27 DEBUG "status": "success", 10:24:27 DEBUG "data": { 10:24:27 DEBUG "resultType": "vector", 10:24:27 DEBUG "result": [ 10:24:27 DEBUG { 10:24:27 DEBUG "metric": { 10:24:27 DEBUG "pod": "ds-cts-2" 10:24:27 DEBUG }, 10:24:27 DEBUG "value": [ 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG [...] 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG "pod": "ds-cts-1" 10:24:27 DEBUG }, 10:24:27 DEBUG "value": [ 10:24:27 DEBUG 1685265867, 10:24:27 DEBUG "0" 10:24:27 DEBUG ] 10:24:27 DEBUG } 10:24:27 DEBUG ] 10:24:27 DEBUG } 10:24:27 DEBUG } 10:24:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265867" 127.0.0.1 - - [28/May/2023 10:24:27] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 10:24:27 INFO [http_cmd]: http status code OK 10:24:27 DEBUG --- status code --- 10:24:27 DEBUG http status code is 200 (expected 200) 10:24:27 DEBUG --- http response --- 10:24:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:24:27 DEBUG ----- output ----- 10:24:27 DEBUG { 10:24:27 DEBUG "status": "success", 10:24:27 DEBUG "data": { 10:24:27 DEBUG "resultType": "vector", 10:24:27 DEBUG "result": [ 10:24:27 DEBUG { 10:24:27 DEBUG "metric": { 10:24:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:24:27 DEBUG }, 10:24:27 DEBUG "value": [ 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG [...] 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:24:27 DEBUG }, 10:24:27 DEBUG "value": [ 10:24:27 DEBUG 1685265867, 10:24:27 DEBUG "1.0293817720832625" 10:24:27 DEBUG ] 10:24:27 DEBUG } 10:24:27 DEBUG ] 10:24:27 DEBUG } 10:24:27 DEBUG } 10:24:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265867" 10:24:27 INFO [http_cmd]: http status code OK 10:24:27 DEBUG --- status code --- 10:24:27 DEBUG http status code is 200 (expected 200) 10:24:27 DEBUG --- http response --- 10:24:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:24:27 DEBUG ----- output ----- 10:24:27 DEBUG { 10:24:27 DEBUG "status": "success", 10:24:27 DEBUG "data": { 10:24:27 DEBUG "resultType": "vector", 10:24:27 DEBUG "result": [ 10:24:27 DEBUG { 10:24:27 DEBUG "metric": { 10:24:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:24:27 DEBUG }, 10:24:27 DEBUG "value": [ 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG [...] 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG 10:24:27 DEBUG "pod": "ds-idrepo-2" 10:24:27 DEBUG }, 10:24:27 DEBUG "value": [ 10:24:27 DEBUG 1685265867, 10:24:27 DEBUG "0" 10:24:27 DEBUG ] 10:24:27 DEBUG } 10:24:27 DEBUG ] 10:24:27 DEBUG } 10:24:27 DEBUG } 127.0.0.1 - - [28/May/2023 10:24:27] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:34] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:35] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 10:24:35 INFO 10:24:35 INFO [loop_until]: kubectl --namespace=xlou top pods 10:24:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:24:35 INFO [loop_until]: OK (rc = 0) 10:24:35 DEBUG --- stdout --- 10:24:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 6Mi am-7849cf7bdb-72wqt 9m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 8m 5769Mi ds-cts-0 9m 460Mi ds-cts-1 8m 414Mi ds-cts-2 10m 428Mi ds-idrepo-0 21m 9905Mi ds-idrepo-1 14m 13713Mi ds-idrepo-2 17m 12551Mi end-user-ui-fdf99b7d7-4gw9n 1m 5Mi idm-58dc667486-gclr8 7m 5341Mi idm-58dc667486-h86fb 7m 4967Mi lodemon-cfd7686d9-6dlt4 32m 68Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 240m 791Mi 10:24:35 DEBUG --- stderr --- 10:24:35 DEBUG 127.0.0.1 - - [28/May/2023 10:24:35] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:36] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 10:24:38 INFO 10:24:38 INFO [loop_until]: kubectl --namespace=xlou top node 10:24:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:24:38 INFO [loop_until]: OK (rc = 0) 10:24:38 DEBUG --- stdout --- 10:24:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 94m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 72m 0% 6659Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 134m 0% 1546Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 76m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 87m 0% 6482Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 81m 0% 6059Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 133m 0% 1356Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 62m 0% 1099Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 55m 0% 1105Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 57m 0% 1081Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 69m 0% 13180Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 80m 0% 10459Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 66m 0% 14337Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 562m 3% 2003Mi 3% 10:24:38 DEBUG --- stderr --- 10:24:38 DEBUG 127.0.0.1 - - [28/May/2023 10:24:43] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:44] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:44] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:45] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:52] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:52] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:53] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:24:54] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:00] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:01] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:02] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:03] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:09] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:10] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:11] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:12] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:18] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:19] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:20] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:21] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265926" 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265926" 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265926" 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265926" 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 DEBUG --- status code --- 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:25:26 DEBUG { 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG { 10:25:26 DEBUG "data": { 10:25:26 DEBUG { 10:25:26 DEBUG { 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG "data": { 10:25:26 DEBUG "result": [ 10:25:26 DEBUG "data": { 10:25:26 DEBUG "data": { 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG { 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG "result": [ 10:25:26 DEBUG "metric": { 10:25:26 DEBUG "result": [ 10:25:26 DEBUG "result": [ 10:25:26 DEBUG { 10:25:26 DEBUG "pod": "ds-idrepo-0" 10:25:26 DEBUG { 10:25:26 DEBUG { 10:25:26 DEBUG "metric": { 10:25:26 DEBUG }, 10:25:26 DEBUG "metric": { 10:25:26 DEBUG "metric": { 10:25:26 DEBUG "pod": "ds-idrepo-0" 10:25:26 DEBUG "value": [ 10:25:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:25:26 DEBUG "instance": "10.142.15.199:9100" 10:25:26 DEBUG }, 10:25:26 DEBUG 10:25:26 DEBUG }, 10:25:26 DEBUG }, 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 10:25:26 DEBUG "value": [ 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG [...] 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG [...] 10:25:26 DEBUG [...] 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG [...] 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG "pod": "ds-idrepo-1" 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG }, 10:25:26 DEBUG 10:25:26 DEBUG "pod": "ds-cts-2" 10:25:26 DEBUG "pod": "ds-cts-1" 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 10:25:26 DEBUG }, 10:25:26 DEBUG }, 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG "instance": "10.142.15.212:9100" 10:25:26 DEBUG "value": [ 10:25:26 DEBUG "value": [ 10:25:26 DEBUG "0" 10:25:26 DEBUG }, 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG ] 10:25:26 DEBUG "value": [ 10:25:26 DEBUG "0" 10:25:26 DEBUG "0" 10:25:26 DEBUG } 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG ] 10:25:26 DEBUG ] 10:25:26 DEBUG ] 10:25:26 DEBUG "0.0006666666666660604" 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG ] 10:25:26 DEBUG ] 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265926" 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG { 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG "data": { 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG "result": [ 10:25:26 DEBUG { 10:25:26 DEBUG "metric": { 10:25:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:25:26 DEBUG }, 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG [...] 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:25:26 DEBUG }, 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG "0" 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265926" 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265926" 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265926" 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG { 10:25:26 DEBUG { 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG "data": { 10:25:26 DEBUG "data": { 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG "result": [ 10:25:26 DEBUG "result": [ 10:25:26 DEBUG { 10:25:26 DEBUG { 10:25:26 DEBUG "metric": { 10:25:26 DEBUG "metric": { 10:25:26 DEBUG "pod": "ds-idrepo-0" 10:25:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:25:26 DEBUG }, 10:25:26 DEBUG }, 10:25:26 DEBUG "value": [ 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG [...] 10:25:26 DEBUG [...] 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG "pod": "ds-cts-1" 10:25:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:25:26 DEBUG }, 10:25:26 DEBUG }, 10:25:26 DEBUG "value": [ 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG "0" 10:25:26 DEBUG "0" 10:25:26 DEBUG ] 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG ] 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG { 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG "data": { 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG "result": [ 10:25:26 DEBUG { 10:25:26 DEBUG "metric": { 10:25:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:25:26 DEBUG }, 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG [...] 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:25:26 DEBUG }, 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG "5589090304" 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG ] 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265926" 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265926" 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265926" 10:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265926" 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 DEBUG --- status code --- 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:26 INFO [http_cmd]: http status code OK 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:26 DEBUG { 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG --- status code --- 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG { 10:25:26 DEBUG http status code is 200 (expected 200) 10:25:26 DEBUG { 10:25:26 DEBUG "data": { 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG --- http response --- 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG "data": { 10:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:26 DEBUG "data": { 10:25:26 DEBUG "result": [ 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG ----- output ----- 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG { 10:25:26 DEBUG "result": [ 10:25:26 DEBUG { 10:25:26 DEBUG "result": [ 10:25:26 DEBUG "metric": { 10:25:26 DEBUG { 10:25:26 DEBUG "status": "success", 10:25:26 DEBUG { 10:25:26 DEBUG "pod": "ds-idrepo-0" 10:25:26 DEBUG "metric": { 10:25:26 DEBUG "data": { 10:25:26 DEBUG "metric": { 10:25:26 DEBUG }, 10:25:26 DEBUG "pod": "ds-idrepo-0" 10:25:26 DEBUG "resultType": "vector", 10:25:26 DEBUG "instance": "10.142.15.199:9100" 10:25:26 DEBUG "value": [ 10:25:26 DEBUG }, 10:25:26 DEBUG "result": [ 10:25:26 DEBUG }, 10:25:26 DEBUG 10:25:26 DEBUG "value": [ 10:25:26 DEBUG { 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG "metric": { 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:25:26 DEBUG 10:25:26 DEBUG [...] 10:25:26 DEBUG 10:25:26 DEBUG }, 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG [...] 10:25:26 DEBUG "value": [ 10:25:26 DEBUG [...] 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG "pod": "ds-idrepo-1" 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG 10:25:26 DEBUG }, 10:25:26 DEBUG "pod": "ds-idrepo-1" 10:25:26 DEBUG [...] 10:25:26 DEBUG "instance": "10.142.15.212:9100" 10:25:26 DEBUG "value": [ 10:25:26 DEBUG }, 10:25:26 DEBUG 10:25:26 DEBUG }, 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG "value": [ 10:25:26 DEBUG 10:25:26 DEBUG "value": [ 10:25:26 DEBUG "0" 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG ] 10:25:26 DEBUG "0" 10:25:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:25:26 DEBUG "26325.1" 10:25:26 DEBUG } 10:25:26 DEBUG ] 10:25:26 DEBUG }, 10:25:26 DEBUG ] 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG "value": [ 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG ] 10:25:26 DEBUG 1685265926, 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG "0.2" 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:26 DEBUG ] 10:25:26 DEBUG } 10:25:26 DEBUG } 10:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265927" 10:25:27 INFO [http_cmd]: http status code OK 10:25:27 DEBUG --- status code --- 10:25:27 DEBUG http status code is 200 (expected 200) 10:25:27 DEBUG --- http response --- 10:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:27 DEBUG ----- output ----- 10:25:27 DEBUG { 10:25:27 DEBUG "status": "success", 10:25:27 DEBUG "data": { 10:25:27 DEBUG "resultType": "vector", 10:25:27 DEBUG "result": [ 10:25:27 DEBUG { 10:25:27 DEBUG "metric": { 10:25:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG [...] 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 1685265927, 10:25:27 DEBUG "0" 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG } 10:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265927" 10:25:27 INFO [http_cmd]: http status code OK 10:25:27 DEBUG --- status code --- 10:25:27 DEBUG http status code is 200 (expected 200) 10:25:27 DEBUG --- http response --- 10:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:25:27 DEBUG ----- output ----- 10:25:27 DEBUG { 10:25:27 DEBUG "status": "success", 10:25:27 DEBUG "data": { 10:25:27 DEBUG "resultType": "vector", 10:25:27 DEBUG "result": [ 10:25:27 DEBUG { 10:25:27 DEBUG "metric": { 10:25:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG [...] 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG "pod": "ds-cts-2" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 1685265927, 10:25:27 DEBUG "0" 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG } 10:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265927" 10:25:27 INFO [http_cmd]: http status code OK 10:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265927" 10:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265927" 10:25:27 DEBUG --- status code --- 10:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265927" 10:25:27 DEBUG http status code is 200 (expected 200) 10:25:27 DEBUG --- http response --- 10:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:25:27 DEBUG ----- output ----- 10:25:27 DEBUG { 10:25:27 DEBUG "status": "success", 10:25:27 DEBUG "data": { 10:25:27 DEBUG "resultType": "vector", 10:25:27 DEBUG "result": [ 10:25:27 DEBUG { 10:25:27 DEBUG "metric": { 10:25:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG [...] 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG "pod": "ds-cts-2" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 1685265927, 10:25:27 DEBUG "752.1392632854677" 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG } 10:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265927" 10:25:27 INFO [http_cmd]: http status code OK 10:25:27 DEBUG --- status code --- 10:25:27 INFO [http_cmd]: http status code OK 10:25:27 DEBUG http status code is 200 (expected 200) 10:25:27 DEBUG --- status code --- 10:25:27 DEBUG --- http response --- 10:25:27 DEBUG http status code is 200 (expected 200) 10:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:25:27 DEBUG --- http response --- 10:25:27 DEBUG ----- output ----- 10:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:27 DEBUG { 10:25:27 INFO [http_cmd]: http status code OK 10:25:27 DEBUG ----- output ----- 10:25:27 DEBUG "status": "success", 10:25:27 DEBUG --- status code --- 10:25:27 DEBUG { 10:25:27 DEBUG "data": { 10:25:27 DEBUG http status code is 200 (expected 200) 10:25:27 DEBUG "status": "success", 10:25:27 DEBUG "resultType": "vector", 10:25:27 DEBUG --- http response --- 10:25:27 DEBUG "data": { 10:25:27 DEBUG "result": [ 10:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:27 DEBUG "resultType": "vector", 10:25:27 DEBUG { 10:25:27 DEBUG ----- output ----- 10:25:27 DEBUG "result": [ 10:25:27 DEBUG "metric": { 10:25:27 DEBUG { 10:25:27 DEBUG { 10:25:27 DEBUG "pod": "ds-cts-2" 10:25:27 DEBUG "status": "success", 10:25:27 DEBUG "metric": { 10:25:27 DEBUG }, 10:25:27 DEBUG "data": { 10:25:27 DEBUG "instance": "10.142.15.199:9100" 10:25:27 DEBUG "value": [ 10:25:27 DEBUG "resultType": "vector", 10:25:27 DEBUG }, 10:25:27 DEBUG 10:25:27 DEBUG "result": [ 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 10:25:27 DEBUG { 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG "metric": { 10:25:27 DEBUG 10:25:27 INFO [http_cmd]: http status code OK 10:25:27 DEBUG [...] 10:25:27 DEBUG "instance": "10.142.15.199:9100" 10:25:27 DEBUG 10:25:27 DEBUG --- status code --- 10:25:27 DEBUG 10:25:27 DEBUG }, 10:25:27 DEBUG [...] 10:25:27 DEBUG http status code is 200 (expected 200) 10:25:27 DEBUG 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 10:25:27 DEBUG --- http response --- 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:27 DEBUG "pod": "ds-cts-1" 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG ----- output ----- 10:25:27 DEBUG }, 10:25:27 DEBUG 10:25:27 DEBUG "instance": "10.142.15.212:9100" 10:25:27 DEBUG { 10:25:27 DEBUG "value": [ 10:25:27 DEBUG [...] 10:25:27 DEBUG }, 10:25:27 DEBUG "status": "success", 10:25:27 DEBUG 1685265927, 10:25:27 DEBUG 10:25:27 DEBUG "value": [ 10:25:27 DEBUG "data": { 10:25:27 DEBUG "0" 10:25:27 DEBUG 10:25:27 DEBUG 1685265927, 10:25:27 DEBUG "resultType": "vector", 10:25:27 DEBUG ] 10:25:27 DEBUG 10:25:27 DEBUG "0.0027666666666666135" 10:25:27 DEBUG "result": [ 10:25:27 DEBUG } 10:25:27 DEBUG "instance": "10.142.15.212:9100" 10:25:27 DEBUG ] 10:25:27 DEBUG { 10:25:27 DEBUG ] 10:25:27 DEBUG }, 10:25:27 DEBUG } 10:25:27 DEBUG "metric": { 10:25:27 DEBUG } 10:25:27 DEBUG "value": [ 10:25:27 DEBUG ] 10:25:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:25:27 DEBUG } 10:25:27 DEBUG 1685265927, 10:25:27 DEBUG } 10:25:27 DEBUG }, 10:25:27 DEBUG "28077.63333333333" 10:25:27 DEBUG } 10:25:27 DEBUG "value": [ 10:25:27 DEBUG ] 10:25:27 DEBUG 10:25:27 DEBUG } 10:25:27 DEBUG 10:25:27 DEBUG ] 10:25:27 DEBUG 10:25:27 DEBUG } 10:25:27 DEBUG [...] 10:25:27 DEBUG } 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 1685265927, 10:25:27 DEBUG "0" 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG } 10:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265927" 10:25:27 INFO [http_cmd]: http status code OK 10:25:27 DEBUG --- status code --- 10:25:27 DEBUG http status code is 200 (expected 200) 10:25:27 DEBUG --- http response --- 10:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:27 DEBUG ----- output ----- 10:25:27 DEBUG { 10:25:27 DEBUG "status": "success", 10:25:27 DEBUG "data": { 10:25:27 DEBUG "resultType": "vector", 10:25:27 DEBUG "result": [ 10:25:27 DEBUG { 10:25:27 DEBUG "metric": { 10:25:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG [...] 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 1685265927, 10:25:27 DEBUG "1.1028829772651532" 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG } 10:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265927" 10:25:27 INFO [http_cmd]: http status code OK 10:25:27 DEBUG --- status code --- 10:25:27 DEBUG http status code is 200 (expected 200) 10:25:27 DEBUG --- http response --- 10:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:25:27 DEBUG ----- output ----- 10:25:27 DEBUG { 10:25:27 DEBUG "status": "success", 10:25:27 DEBUG "data": { 10:25:27 DEBUG "resultType": "vector", 10:25:27 DEBUG "result": [ 10:25:27 DEBUG { 10:25:27 DEBUG "metric": { 10:25:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG [...] 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG 10:25:27 DEBUG "pod": "ds-cts-2" 10:25:27 DEBUG }, 10:25:27 DEBUG "value": [ 10:25:27 DEBUG 1685265927, 10:25:27 DEBUG "0.007675570200888453" 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG ] 10:25:27 DEBUG } 10:25:27 DEBUG } 127.0.0.1 - - [28/May/2023 10:25:27] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:28] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:29] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:30] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 10:25:35 INFO 10:25:35 INFO [loop_until]: kubectl --namespace=xlou top pods 10:25:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:25:35 INFO [loop_until]: OK (rc = 0) 10:25:35 DEBUG --- stdout --- 10:25:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 6Mi am-7849cf7bdb-72wqt 8m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 8m 461Mi ds-cts-1 8m 414Mi ds-cts-2 9m 428Mi ds-idrepo-0 19m 9905Mi ds-idrepo-1 16m 13713Mi ds-idrepo-2 17m 12551Mi end-user-ui-fdf99b7d7-4gw9n 1m 5Mi idm-58dc667486-gclr8 7m 5341Mi idm-58dc667486-h86fb 7m 4966Mi lodemon-cfd7686d9-6dlt4 33m 69Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 1049m 987Mi 10:25:35 DEBUG --- stderr --- 10:25:35 DEBUG 127.0.0.1 - - [28/May/2023 10:25:36] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:37] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:38] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 10:25:38 INFO 10:25:38 INFO [loop_until]: kubectl --namespace=xlou top node 10:25:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:25:38 INFO [loop_until]: OK (rc = 0) 10:25:38 DEBUG --- stdout --- 10:25:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 96m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6656Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 138m 0% 1542Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6822Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 85m 0% 6481Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 81m 0% 6056Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 133m 0% 1364Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1095Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 58m 0% 1106Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 57m 0% 1080Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 68m 0% 13173Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 79m 0% 10459Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14337Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 1061m 6% 1927Mi 3% 10:25:38 DEBUG --- stderr --- 10:25:38 DEBUG 127.0.0.1 - - [28/May/2023 10:25:39] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:39] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:46] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:47] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:48] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:48] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:49] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:56] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:57] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:57] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:58] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:25:59] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:05] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:06] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:07] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:08] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:09] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:15] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:16] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:17] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:18] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:18] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:25] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:26] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 10:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265986" 10:26:26 INFO [http_cmd]: http status code OK 10:26:26 DEBUG --- status code --- 10:26:26 DEBUG http status code is 200 (expected 200) 10:26:26 DEBUG --- http response --- 10:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:26 DEBUG ----- output ----- 10:26:26 DEBUG { 10:26:26 DEBUG "status": "success", 10:26:26 DEBUG "data": { 10:26:26 DEBUG "resultType": "vector", 10:26:26 DEBUG "result": [ 10:26:26 DEBUG { 10:26:26 DEBUG "metric": { 10:26:26 DEBUG "pod": "ds-idrepo-0" 10:26:26 DEBUG }, 10:26:26 DEBUG "value": [ 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG [...] 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG "pod": "ds-idrepo-1" 10:26:26 DEBUG }, 10:26:26 DEBUG "value": [ 10:26:26 DEBUG 1685265986, 10:26:26 DEBUG "0" 10:26:26 DEBUG ] 10:26:26 DEBUG } 10:26:26 DEBUG ] 10:26:26 DEBUG } 10:26:26 DEBUG } 10:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265986" 10:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265986" 10:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265986" 10:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265986" 10:26:26 INFO [http_cmd]: http status code OK 127.0.0.1 - - [28/May/2023 10:26:26] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 10:26:26 INFO [http_cmd]: http status code OK 10:26:26 INFO [http_cmd]: http status code OK 10:26:26 DEBUG --- status code --- 10:26:26 INFO [http_cmd]: http status code OK 10:26:26 DEBUG --- status code --- 10:26:26 DEBUG --- status code --- 10:26:26 DEBUG http status code is 200 (expected 200) 10:26:26 DEBUG --- status code --- 10:26:26 DEBUG http status code is 200 (expected 200) 10:26:26 DEBUG http status code is 200 (expected 200) 10:26:26 DEBUG --- http response --- 10:26:26 DEBUG http status code is 200 (expected 200) 10:26:26 DEBUG --- http response --- 10:26:26 DEBUG --- http response --- 10:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:26 DEBUG --- http response --- 10:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:26 DEBUG ----- output ----- 10:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:26 DEBUG ----- output ----- 10:26:26 DEBUG ----- output ----- 10:26:26 DEBUG { 10:26:26 DEBUG ----- output ----- 10:26:26 DEBUG { 10:26:26 DEBUG { 10:26:26 DEBUG "status": "success", 10:26:26 DEBUG { 10:26:26 DEBUG "status": "success", 10:26:26 DEBUG "status": "success", 10:26:26 DEBUG "data": { 10:26:26 DEBUG "status": "success", 10:26:26 DEBUG "data": { 10:26:26 DEBUG "data": { 10:26:26 DEBUG "resultType": "vector", 10:26:26 DEBUG "data": { 10:26:26 DEBUG "resultType": "vector", 10:26:26 DEBUG "resultType": "vector", 10:26:26 DEBUG "result": [ 10:26:26 DEBUG "resultType": "vector", 10:26:26 DEBUG "result": [ 10:26:26 DEBUG "result": [ 10:26:26 DEBUG { 10:26:26 DEBUG "result": [ 10:26:26 DEBUG { 10:26:26 DEBUG { 10:26:26 DEBUG "metric": { 10:26:26 DEBUG { 10:26:26 DEBUG "metric": { 10:26:26 DEBUG "metric": { 10:26:26 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:26:26 DEBUG "metric": { 10:26:26 DEBUG "pod": "ds-idrepo-0" 10:26:26 DEBUG "instance": "10.142.15.199:9100" 10:26:26 DEBUG }, 10:26:26 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:26:26 DEBUG }, 10:26:26 DEBUG }, 10:26:26 DEBUG "value": [ 10:26:26 DEBUG }, 10:26:26 DEBUG "value": [ 10:26:26 DEBUG "value": [ 10:26:26 DEBUG 10:26:26 DEBUG "value": [ 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG [...] 10:26:26 DEBUG 10:26:26 DEBUG [...] 10:26:26 DEBUG [...] 10:26:26 DEBUG 10:26:26 DEBUG [...] 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG 10:26:26 DEBUG "pod": "ds-idrepo-2" 10:26:26 DEBUG 10:26:26 DEBUG "pod": "ds-cts-1" 10:26:26 DEBUG "instance": "10.142.15.212:9100" 10:26:26 DEBUG }, 10:26:26 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:26:26 DEBUG }, 10:26:26 DEBUG }, 10:26:26 DEBUG "value": [ 10:26:26 DEBUG }, 10:26:26 DEBUG "value": [ 10:26:26 DEBUG "value": [ 10:26:26 DEBUG 1685265986, 10:26:26 DEBUG "value": [ 10:26:26 DEBUG 1685265986, 10:26:26 DEBUG 1685265986, 10:26:26 DEBUG "0.5984854466024527" 10:26:26 DEBUG 1685265986, 10:26:26 DEBUG "0" 10:26:26 DEBUG "0.0006666666666660604" 10:26:26 DEBUG ] 10:26:26 DEBUG "0" 10:26:26 DEBUG ] 10:26:26 DEBUG ] 10:26:26 DEBUG } 10:26:26 DEBUG ] 10:26:26 DEBUG } 10:26:26 DEBUG } 10:26:26 DEBUG ] 10:26:26 DEBUG } 10:26:26 DEBUG ] 10:26:26 DEBUG ] 10:26:26 DEBUG } 10:26:26 DEBUG ] 10:26:26 DEBUG } 10:26:26 DEBUG } 10:26:26 DEBUG } 10:26:26 DEBUG } 10:26:26 DEBUG } 10:26:26 DEBUG } 10:26:26 DEBUG } 10:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265986" 10:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265986" 10:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265986" 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "data": { 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "result": [ 10:26:27 DEBUG "result": [ 10:26:27 DEBUG { 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:26:27 DEBUG "pod": "ds-idrepo-0" 10:26:27 DEBUG }, 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG "value": [ 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685265987" 10:26:27 DEBUG 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG { 10:26:27 DEBUG 10:26:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "pod": "ds-cts-1" 10:26:27 DEBUG }, 10:26:27 DEBUG "data": { 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265986, 10:26:27 DEBUG "result": [ 10:26:27 DEBUG 1685265986, 10:26:27 DEBUG "0" 10:26:27 DEBUG { 10:26:27 DEBUG "0" 10:26:27 DEBUG ] 10:26:27 DEBUG "metric": { 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG "instance": "10.142.15.199:9100" 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG }, 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG "value": [ 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 DEBUG 10:26:27 DEBUG } 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "instance": "10.142.15.212:9100" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265986, 10:26:27 DEBUG "27983" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "result": [ 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG "5590224896" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265987" 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265987" 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265987" 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:27 DEBUG { 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG { 10:26:27 DEBUG { 10:26:27 DEBUG "data": { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "data": { 10:26:27 DEBUG "result": [ 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG { 10:26:27 DEBUG "result": [ 10:26:27 DEBUG "result": [ 10:26:27 DEBUG "metric": { 10:26:27 DEBUG { 10:26:27 DEBUG { 10:26:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG }, 10:26:27 DEBUG "pod": "ds-idrepo-0" 10:26:27 DEBUG "pod": "ds-idrepo-0" 10:26:27 DEBUG "value": [ 10:26:27 DEBUG }, 10:26:27 DEBUG }, 10:26:27 DEBUG 10:26:27 DEBUG "value": [ 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG }, 10:26:27 DEBUG "pod": "ds-idrepo-1" 10:26:27 DEBUG "pod": "ds-idrepo-1" 10:26:27 DEBUG "value": [ 10:26:27 DEBUG }, 10:26:27 DEBUG }, 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG "value": [ 10:26:27 DEBUG "0.2" 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG ] 10:26:27 DEBUG "0" 10:26:27 DEBUG "0" 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG ] 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265987" 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265987" 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "result": [ 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG "0" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265987" 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "result": [ 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "instance": "10.142.15.199:9100" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "instance": "10.142.15.212:9100" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG "28204.933333333334" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "result": [ 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "pod": "ds-idrepo-2" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG "0" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265987" 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685265987" 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "result": [ 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265987" 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "pod": "ds-idrepo-2" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG "356116523.2528027" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265987" 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "result": [ 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "instance": "10.142.15.199:9100" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "instance": "10.142.15.212:9100" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG "0.00280000000000058" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG { 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG "data": { 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:27 DEBUG "result": [ 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "pod": "ds-cts-2" 10:26:27 DEBUG "data": { 10:26:27 DEBUG }, 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "value": [ 10:26:27 DEBUG "result": [ 10:26:27 DEBUG 10:26:27 DEBUG { 10:26:27 DEBUG 10:26:27 DEBUG "metric": { 10:26:27 DEBUG 10:26:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:26:27 DEBUG [...] 10:26:27 DEBUG }, 10:26:27 DEBUG 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "pod": "ds-cts-1" 10:26:27 DEBUG 10:26:27 DEBUG }, 10:26:27 DEBUG [...] 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG 10:26:27 DEBUG "0" 10:26:27 DEBUG 10:26:27 DEBUG ] 10:26:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:26:27 DEBUG } 10:26:27 DEBUG }, 10:26:27 DEBUG ] 10:26:27 DEBUG "value": [ 10:26:27 DEBUG } 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG } 10:26:27 DEBUG "0" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685265987" 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "result": [ 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG "0.6127128711883765" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 10:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685265987" 10:26:27 INFO [http_cmd]: http status code OK 10:26:27 DEBUG --- status code --- 10:26:27 DEBUG http status code is 200 (expected 200) 10:26:27 DEBUG --- http response --- 10:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:26:27 DEBUG ----- output ----- 10:26:27 DEBUG { 10:26:27 DEBUG "status": "success", 10:26:27 DEBUG "data": { 10:26:27 DEBUG "resultType": "vector", 10:26:27 DEBUG "result": [ 10:26:27 DEBUG { 10:26:27 DEBUG "metric": { 10:26:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG [...] 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG 10:26:27 DEBUG "pod": "ds-idrepo-2" 10:26:27 DEBUG }, 10:26:27 DEBUG "value": [ 10:26:27 DEBUG 1685265987, 10:26:27 DEBUG "0.01081726767132267" 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG ] 10:26:27 DEBUG } 10:26:27 DEBUG } 127.0.0.1 - - [28/May/2023 10:26:27] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:28] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:35] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 10:26:35 INFO 10:26:35 INFO [loop_until]: kubectl --namespace=xlou top pods 10:26:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:26:35 INFO [loop_until]: OK (rc = 0) 10:26:35 DEBUG --- stdout --- 10:26:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 6Mi am-7849cf7bdb-72wqt 9m 5743Mi am-7849cf7bdb-cwb58 8m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 9m 460Mi ds-cts-1 9m 414Mi ds-cts-2 9m 428Mi ds-idrepo-0 21m 9904Mi ds-idrepo-1 14m 13713Mi ds-idrepo-2 17m 12551Mi end-user-ui-fdf99b7d7-4gw9n 1m 5Mi idm-58dc667486-gclr8 8m 5341Mi idm-58dc667486-h86fb 7m 4966Mi lodemon-cfd7686d9-6dlt4 33m 68Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 1005m 1551Mi 10:26:35 DEBUG --- stderr --- 10:26:35 DEBUG 127.0.0.1 - - [28/May/2023 10:26:35] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:36] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:37] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:38] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 10:26:38 INFO 10:26:38 INFO [loop_until]: kubectl --namespace=xlou top node 10:26:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:26:38 INFO [loop_until]: OK (rc = 0) 10:26:38 DEBUG --- stdout --- 10:26:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 95m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 70m 0% 6654Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 130m 0% 1545Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 77m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 84m 0% 6485Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 82m 0% 6054Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 138m 0% 1354Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 59m 0% 1101Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1107Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 59m 0% 1081Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 69m 0% 13177Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 82m 0% 10462Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 65m 0% 14338Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 1082m 6% 2478Mi 4% 10:26:38 DEBUG --- stderr --- 10:26:38 DEBUG 127.0.0.1 - - [28/May/2023 10:26:44] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:45] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:46] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:47] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:48] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:55] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:55] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:56] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:57] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:26:58] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:04] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:05] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:06] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:07] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:08] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:08] "GET /monitoring/average?start_time=23-05-28_08:35:04&stop_time=23-05-28_08:57:29 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:15] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:16] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:17] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:17] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:18] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:19] "GET /monitoring/average?start_time=23-05-28_08:35:04&stop_time=23-05-28_08:57:29 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:25] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:26] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 10:27:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266046" 10:27:26 INFO [http_cmd]: http status code OK 10:27:26 DEBUG --- status code --- 10:27:26 DEBUG http status code is 200 (expected 200) 10:27:26 DEBUG --- http response --- 10:27:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:26 DEBUG ----- output ----- 10:27:26 DEBUG { 10:27:26 DEBUG "status": "success", 10:27:26 DEBUG "data": { 10:27:26 DEBUG "resultType": "vector", 10:27:26 DEBUG "result": [ 10:27:26 DEBUG { 10:27:26 DEBUG "metric": { 10:27:26 DEBUG "pod": "ds-idrepo-0" 10:27:26 DEBUG }, 10:27:26 DEBUG "value": [ 10:27:26 DEBUG 10:27:26 DEBUG 10:27:26 DEBUG 10:27:26 DEBUG [...] 10:27:26 DEBUG 10:27:26 DEBUG 10:27:26 DEBUG 10:27:26 DEBUG "pod": "ds-idrepo-1" 10:27:26 DEBUG }, 10:27:26 DEBUG "value": [ 10:27:26 DEBUG 1685266046, 10:27:26 DEBUG "0" 10:27:26 DEBUG ] 10:27:26 DEBUG } 10:27:26 DEBUG ] 10:27:26 DEBUG } 10:27:26 DEBUG } 10:27:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266046" 10:27:26 INFO [http_cmd]: http status code OK 10:27:26 DEBUG --- status code --- 10:27:26 DEBUG http status code is 200 (expected 200) 10:27:26 DEBUG --- http response --- 10:27:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:27:26 DEBUG ----- output ----- 10:27:26 DEBUG { 10:27:26 DEBUG "status": "success", 10:27:26 DEBUG "data": { 10:27:26 DEBUG "resultType": "vector", 10:27:26 DEBUG "result": [ 10:27:26 DEBUG { 10:27:26 DEBUG "metric": { 10:27:26 DEBUG "pod": "ds-idrepo-0" 10:27:26 DEBUG }, 10:27:26 DEBUG "value": [ 10:27:26 DEBUG 10:27:26 DEBUG 10:27:26 DEBUG 10:27:26 DEBUG [...] 10:27:26 DEBUG 10:27:26 DEBUG 10:27:26 DEBUG 10:27:26 DEBUG "pod": "ds-cts-1" 10:27:26 DEBUG }, 10:27:26 DEBUG "value": [ 10:27:26 DEBUG 1685266046, 10:27:26 DEBUG "0" 10:27:26 DEBUG ] 10:27:26 DEBUG } 10:27:26 DEBUG ] 10:27:26 DEBUG } 10:27:26 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "instance": "10.142.15.199:9100" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "instance": "10.142.15.212:9100" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "0.0010000000000066696" 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG { 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "data": { 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG "result": [ 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG { 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:27:27 DEBUG "pod": "ds-idrepo-0" 10:27:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:27:27 DEBUG }, 10:27:27 DEBUG }, 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG [...] 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:27:27 DEBUG "pod": "ds-cts-1" 10:27:27 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:27:27 DEBUG }, 10:27:27 DEBUG }, 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "0" 10:27:27 DEBUG "0" 10:27:27 DEBUG "0.5258639395512027" 10:27:27 DEBUG ] 10:27:27 DEBUG ] 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG ] 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "pod": "ds-idrepo-0" 10:27:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:27:27 DEBUG }, 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "pod": "ds-idrepo-1" 10:27:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:27:27 DEBUG }, 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "0" 10:27:27 DEBUG "0" 10:27:27 DEBUG ] 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685266047" 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:27 DEBUG { 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG { 10:27:27 DEBUG "data": { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "result": [ 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG { 10:27:27 DEBUG "result": [ 10:27:27 DEBUG "metric": { 10:27:27 DEBUG { 10:27:27 DEBUG "instance": "10.142.15.199:9100" 10:27:27 DEBUG "metric": { 10:27:27 DEBUG }, 10:27:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:27:27 DEBUG "value": [ 10:27:27 DEBUG }, 10:27:27 DEBUG 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "instance": "10.142.15.212:9100" 10:27:27 DEBUG 10:27:27 DEBUG }, 10:27:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-b0st" 10:27:27 DEBUG "value": [ 10:27:27 DEBUG }, 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "26537.266666666666" 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG ] 10:27:27 DEBUG "5591711744" 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:27 DEBUG { 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG { 10:27:27 DEBUG "data": { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "result": [ 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG { 10:27:27 DEBUG "result": [ 10:27:27 DEBUG "metric": { 10:27:27 DEBUG { 10:27:27 DEBUG "pod": "ds-idrepo-0" 10:27:27 DEBUG "metric": { 10:27:27 DEBUG }, 10:27:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:27:27 DEBUG "value": [ 10:27:27 DEBUG }, 10:27:27 DEBUG 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "pod": "ds-idrepo-1" 10:27:27 DEBUG 10:27:27 DEBUG }, 10:27:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:27:27 DEBUG "value": [ 10:27:27 DEBUG }, 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "0" 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG ] 10:27:27 DEBUG "0.2" 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1685266047" 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "86626510.02686414" 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG { 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG "data": { 10:27:27 DEBUG { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG "data": { 10:27:27 DEBUG { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "result": [ 10:27:27 DEBUG "instance": "10.142.15.199:9100" 10:27:27 DEBUG { 10:27:27 DEBUG }, 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "instance": "10.142.15.199:9100" 10:27:27 DEBUG 10:27:27 DEBUG }, 10:27:27 DEBUG 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "instance": "10.142.15.212:9100" 10:27:27 DEBUG 10:27:27 DEBUG }, 10:27:27 DEBUG 10:27:27 DEBUG "value": [ 10:27:27 DEBUG "instance": "10.142.15.212:9100" 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG }, 10:27:27 DEBUG "0.0027666666666663767" 10:27:27 DEBUG "value": [ 10:27:27 DEBUG ] 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG } 10:27:27 DEBUG "28163.666666666668" 10:27:27 DEBUG ] 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "0" 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "0" 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "pod": "ds-cts-2" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "pod": "ds-cts-1" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "0" 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "pod": "am-7849cf7bdb-vcl2f" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "pod": "am-7849cf7bdb-cwb58" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "0" 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-bqd0" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-ndrl" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "1.0380448434723144" 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 10:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1685266047" 10:27:27 INFO [http_cmd]: http status code OK 10:27:27 DEBUG --- status code --- 10:27:27 DEBUG http status code is 200 (expected 200) 10:27:27 DEBUG --- http response --- 10:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:27:27 DEBUG ----- output ----- 10:27:27 DEBUG { 10:27:27 DEBUG "status": "success", 10:27:27 DEBUG "data": { 10:27:27 DEBUG "resultType": "vector", 10:27:27 DEBUG "result": [ 10:27:27 DEBUG { 10:27:27 DEBUG "metric": { 10:27:27 DEBUG "pod": "admin-ui-7c7859f797-fzrqk" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG [...] 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG 10:27:27 DEBUG "pod": "overseer-0-64c8c4b58-d9cvt" 10:27:27 DEBUG }, 10:27:27 DEBUG "value": [ 10:27:27 DEBUG 1685266047, 10:27:27 DEBUG "1.2947157703441454" 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG ] 10:27:27 DEBUG } 10:27:27 DEBUG } 127.0.0.1 - - [28/May/2023 10:27:27] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:28] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:29] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:30] "GET /monitoring/average?start_time=23-05-28_08:35:04&stop_time=23-05-28_08:57:29 HTTP/1.1" 200 - 10:27:35 INFO 10:27:35 INFO [loop_until]: kubectl --namespace=xlou top pods 10:27:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:27:35 INFO [loop_until]: OK (rc = 0) 10:27:35 DEBUG --- stdout --- 10:27:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7c7859f797-fzrqk 1m 6Mi am-7849cf7bdb-72wqt 9m 5743Mi am-7849cf7bdb-cwb58 7m 5816Mi am-7849cf7bdb-vcl2f 7m 5769Mi ds-cts-0 8m 460Mi ds-cts-1 8m 414Mi ds-cts-2 8m 428Mi ds-idrepo-0 20m 9905Mi ds-idrepo-1 14m 13713Mi ds-idrepo-2 16m 12551Mi end-user-ui-fdf99b7d7-4gw9n 1m 5Mi idm-58dc667486-gclr8 7m 5341Mi idm-58dc667486-h86fb 7m 4966Mi lodemon-cfd7686d9-6dlt4 36m 68Mi login-ui-7fdfb4d89f-szrsp 1m 3Mi overseer-0-64c8c4b58-d9cvt 640m 1597Mi 10:27:35 DEBUG --- stderr --- 10:27:35 DEBUG 127.0.0.1 - - [28/May/2023 10:27:36] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:37] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:38] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 10:27:38 INFO 10:27:38 INFO [loop_until]: kubectl --namespace=xlou top node 10:27:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:27:38 INFO [loop_until]: OK (rc = 0) 10:27:38 DEBUG --- stdout --- 10:27:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-2f0c 92m 0% 6850Mi 11% gke-xlou-cdm-default-pool-f05840a3-3jkl 69m 0% 6653Mi 11% gke-xlou-cdm-default-pool-f05840a3-698n 123m 0% 1545Mi 2% gke-xlou-cdm-default-pool-f05840a3-6l31 78m 0% 6823Mi 11% gke-xlou-cdm-default-pool-f05840a3-97zw 79m 0% 6483Mi 11% gke-xlou-cdm-default-pool-f05840a3-b0st 82m 0% 6057Mi 10% gke-xlou-cdm-default-pool-f05840a3-bqd0 144m 0% 1354Mi 2% gke-xlou-cdm-ds-32e4dcb1-83pt 61m 0% 1097Mi 1% gke-xlou-cdm-ds-32e4dcb1-92tk 57m 0% 1106Mi 1% gke-xlou-cdm-ds-32e4dcb1-dpl5 58m 0% 1082Mi 1% gke-xlou-cdm-ds-32e4dcb1-tgjd 68m 0% 13176Mi 22% gke-xlou-cdm-ds-32e4dcb1-zf1k 77m 0% 10459Mi 17% gke-xlou-cdm-ds-32e4dcb1-zg6n 67m 0% 14336Mi 24% gke-xlou-cdm-frontend-a8771548-ndrl 1054m 6% 2596Mi 4% 10:27:38 DEBUG --- stderr --- 10:27:38 DEBUG 127.0.0.1 - - [28/May/2023 10:27:38] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:39] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:40] "GET /monitoring/average?start_time=23-05-28_08:35:04&stop_time=23-05-28_08:57:29 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:47] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:47] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:48] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:49] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:50] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:51] "GET /monitoring/average?start_time=23-05-28_08:35:04&stop_time=23-05-28_08:57:29 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:57] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:58] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:27:59] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:00] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:00] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:01] "GET /monitoring/average?start_time=23-05-28_08:35:04&stop_time=23-05-28_08:57:29 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:08] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:09] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:09] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:10] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:11] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:12] "GET /monitoring/average?start_time=23-05-28_08:35:04&stop_time=23-05-28_08:57:29 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:19] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:19] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:20] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:21] "GET /monitoring/average?start_time=23-05-28_07:50:17&stop_time=23-05-28_08:12:41 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:22] "GET /monitoring/average?start_time=23-05-28_08:12:41&stop_time=23-05-28_08:35:04 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:23] "GET /monitoring/average?start_time=23-05-28_08:35:04&stop_time=23-05-28_08:57:29 HTTP/1.1" 200 - 10:28:24 INFO Finished: True 10:28:24 INFO Waiting for threads to register finish flag 10:28:38 INFO Done. Have a nice day! :) 127.0.0.1 - - [28/May/2023 10:28:38] "GET /monitoring/stop HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:38] "GET /monitoring/average?start_time=23-05-28_06:42:54&stop_time=23-05-28_07:05:24 HTTP/1.1" 200 - 10:28:45 INFO File /tmp/lodemon_data-23-05-28_06:58:37/am_cts_reaper_cache_size.json does not exist. Skipping... 10:28:45 INFO File /tmp/lodemon_data-23-05-28_06:58:37/am_cts_reaper_search_seconds_total.json does not exist. Skipping... 10:28:45 INFO File /tmp/lodemon_data-23-05-28_06:58:37/ds_replication_replica_replayed_updates_conflicts_resolved.json does not exist. Skipping... 10:28:45 INFO File /tmp/lodemon_data-23-05-28_06:58:37/node_disk_read_bytes_total.json does not exist. Skipping... 10:28:45 INFO File /tmp/lodemon_data-23-05-28_06:58:37/node_disk_written_bytes_total.json does not exist. Skipping... 10:28:45 INFO File /tmp/lodemon_data-23-05-28_06:58:37/ds_backend_entry_count.json does not exist. Skipping... 127.0.0.1 - - [28/May/2023 10:28:52] "GET /monitoring/process HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:52] "GET /monitoring/average?start_time=23-05-28_07:05:24&stop_time=23-05-28_07:27:53 HTTP/1.1" 200 - 127.0.0.1 - - [28/May/2023 10:28:54] "GET /monitoring/average?start_time=23-05-28_07:27:53&stop_time=23-05-28_07:50:17 HTTP/1.1" 200 -