==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-75f677b97d-6b58v Namespace: xlou Priority: 0 Node: gke-xlou-cdm-default-pool-f05840a3-s6rh/10.142.0.104 Start Time: Sun, 11 Jun 2023 18:41:52 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=75f677b97d skaffold.dev/run-id=94be0a57-447e-4f3b-be65-9f2b05fd88b8 Annotations: Status: Running IP: 10.106.46.40 IPs: IP: 10.106.46.40 Controlled By: ReplicaSet/lodemon-75f677b97d Containers: lodemon: Container ID: containerd://6e264a71989a1d7f7c5e6fc3128dddfc766a716292f64473998d7b37e7561a35 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:0e43c68f3b80d8ecf0fafa39dba0b367e5e9774eafcea31fa3e89306a12df089 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sun, 11 Jun 2023 18:41:53 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qbsh5 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-qbsh5: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 08:05:52 INFO 08:05:52 INFO [loop_until]: kubectl --namespace=xlou top pods 08:05:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:05:52 INFO [loop_until]: OK (rc = 0) 08:05:52 DEBUG --- stdout --- 08:05:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 4Mi am-7849cf7bdb-bd9j2 274m 6118Mi am-7849cf7bdb-npvzp 307m 6227Mi am-7849cf7bdb-p4fns 284m 6283Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 6332m 13826Mi ds-idrepo-1 4520m 13648Mi ds-idrepo-2 3473m 13653Mi end-user-ui-5c589c665b-tgm8n 1m 3Mi idm-58dc667486-ml9mc 1921m 5465Mi idm-58dc667486-vrqsf 2109m 5526Mi lodemon-75f677b97d-6b58v 7m 71Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 688m 1704Mi 08:05:52 DEBUG --- stderr --- 08:05:52 DEBUG 08:06:04 INFO 08:06:04 INFO [loop_until]: kubectl --namespace=xlou top node 08:06:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:06:04 INFO [loop_until]: OK (rc = 0) 08:06:04 DEBUG --- stdout --- 08:06:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2239m 14% 6421Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 336m 2% 7158Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 332m 2% 7051Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 2247m 14% 6415Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 346m 2% 7374Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 1478m 9% 3042Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 86m 0% 1207Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 73m 0% 1123Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1151Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 3651m 22% 14632Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 4732m 29% 14607Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 69m 0% 1051Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 5923m 37% 14670Mi 25% gke-xlou-cdm-frontend-a8771548-h3n6 723m 4% 2581Mi 4% 08:06:04 DEBUG --- stderr --- 08:06:04 DEBUG 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553578" 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553578" 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553578" 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553578" 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553578" 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553578" 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG { 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "data": { 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "result": [ 08:06:18 DEBUG { 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG [...] 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG }, 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG "value": [ 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG "0" 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG ] 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:06:18 DEBUG } 08:06:18 DEBUG { 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG ] 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG { 08:06:18 DEBUG { 08:06:18 DEBUG { 08:06:18 DEBUG } 08:06:18 DEBUG "data": { 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG } 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG "data": { 08:06:18 DEBUG "data": { 08:06:18 DEBUG "data": { 08:06:18 DEBUG "result": [ 08:06:18 DEBUG { 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG { 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "result": [ 08:06:18 DEBUG "result": [ 08:06:18 DEBUG "result": [ 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "data": { 08:06:18 DEBUG { 08:06:18 DEBUG { 08:06:18 DEBUG { 08:06:18 DEBUG "pod": "ds-idrepo-2" 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "metric": { 08:06:18 DEBUG }, 08:06:18 DEBUG "result": [ 08:06:18 DEBUG "pod": "ds-cts-2" 08:06:18 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:06:18 DEBUG "instance": "10.142.0.100:9100" 08:06:18 DEBUG "value": [ 08:06:18 DEBUG { 08:06:18 DEBUG }, 08:06:18 DEBUG }, 08:06:18 DEBUG }, 08:06:18 DEBUG 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "value": [ 08:06:18 DEBUG "value": [ 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG "pod": "ds-idrepo-2" 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG }, 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG [...] 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG [...] 08:06:18 DEBUG [...] 08:06:18 DEBUG [...] 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG "pod": "ds-idrepo-1" 08:06:18 DEBUG [...] 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG }, 08:06:18 DEBUG 08:06:18 DEBUG "pod": "ds-idrepo-1" 08:06:18 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:06:18 DEBUG "instance": "10.142.0.99:9100" 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG }, 08:06:18 DEBUG }, 08:06:18 DEBUG }, 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG 08:06:18 DEBUG "value": [ 08:06:18 DEBUG "value": [ 08:06:18 DEBUG "value": [ 08:06:18 DEBUG "40129.86666666667" 08:06:18 DEBUG "pod": "ds-idrepo-1" 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG ] 08:06:18 DEBUG }, 08:06:18 DEBUG "1279.5333333333333" 08:06:18 DEBUG "0" 08:06:18 DEBUG "3892114.333333333" 08:06:18 DEBUG } 08:06:18 DEBUG "value": [ 08:06:18 DEBUG ] 08:06:18 DEBUG ] 08:06:18 DEBUG ] 08:06:18 DEBUG ] 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 DEBUG "6307.5" 08:06:18 DEBUG ] 08:06:18 DEBUG ] 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553578" 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG { 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "data": { 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "result": [ 08:06:18 DEBUG { 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG [...] 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG "0" 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553578" 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553578" 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG { 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "data": { 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "result": [ 08:06:18 DEBUG { 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "pod": "ds-cts-2" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG [...] 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG "pod": "ds-idrepo-1" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG "0" 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG ] 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG } 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:06:18 DEBUG ] 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG } 08:06:18 DEBUG { 08:06:18 DEBUG } 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "data": { 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "result": [ 08:06:18 DEBUG { 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG [...] 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG "0" 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553578" 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686553578" 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG { 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "data": { 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "result": [ 08:06:18 DEBUG { 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "pod": "ds-idrepo-2" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG [...] 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG "pod": "ds-idrepo-1" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 1686553578, 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG "33179.03333333333" 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG ] 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG } 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG ] 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:06:18 DEBUG } 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG } 08:06:18 DEBUG { 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "data": { 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "result": [ 08:06:18 DEBUG { 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG [...] 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG "0.6745112063426809" 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553578" 08:06:18 INFO [http_cmd]: http status code OK 08:06:18 DEBUG --- status code --- 08:06:18 DEBUG http status code is 200 (expected 200) 08:06:18 DEBUG --- http response --- 08:06:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:06:18 DEBUG ----- output ----- 08:06:18 DEBUG { 08:06:18 DEBUG "status": "success", 08:06:18 DEBUG "data": { 08:06:18 DEBUG "resultType": "vector", 08:06:18 DEBUG "result": [ 08:06:18 DEBUG { 08:06:18 DEBUG "metric": { 08:06:18 DEBUG "instance": "10.142.0.100:9100" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG [...] 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG 08:06:18 DEBUG "instance": "10.142.0.99:9100" 08:06:18 DEBUG }, 08:06:18 DEBUG "value": [ 08:06:18 DEBUG 1686553578, 08:06:18 DEBUG "0.0013333333333333938" 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG ] 08:06:18 DEBUG } 08:06:18 DEBUG } 08:06:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553579" 08:06:19 INFO [http_cmd]: http status code OK 08:06:19 DEBUG --- status code --- 08:06:19 DEBUG http status code is 200 (expected 200) 08:06:19 DEBUG --- http response --- 08:06:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:06:19 DEBUG ----- output ----- 08:06:19 DEBUG { 08:06:19 DEBUG "status": "success", 08:06:19 DEBUG "data": { 08:06:19 DEBUG "resultType": "vector", 08:06:19 DEBUG "result": [ 08:06:19 DEBUG { 08:06:19 DEBUG "metric": { 08:06:19 DEBUG "instance": "10.142.0.100:9100" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG [...] 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG "instance": "10.142.0.99:9100" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 1686553579, 08:06:19 DEBUG "0.0022000000000010306" 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG } 08:06:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553579" 08:06:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553579" 08:06:19 INFO [http_cmd]: http status code OK 08:06:19 DEBUG --- status code --- 08:06:19 DEBUG http status code is 200 (expected 200) 08:06:19 DEBUG --- http response --- 08:06:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:06:19 DEBUG ----- output ----- 08:06:19 DEBUG { 08:06:19 DEBUG "status": "success", 08:06:19 DEBUG "data": { 08:06:19 DEBUG "resultType": "vector", 08:06:19 DEBUG "result": [ 08:06:19 DEBUG { 08:06:19 DEBUG "metric": { 08:06:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG [...] 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 1686553579, 08:06:19 DEBUG "0.2" 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG } 08:06:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686553579" 08:06:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553579" 08:06:19 INFO [http_cmd]: http status code OK 08:06:19 DEBUG --- status code --- 08:06:19 DEBUG http status code is 200 (expected 200) 08:06:19 DEBUG --- http response --- 08:06:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:06:19 DEBUG ----- output ----- 08:06:19 DEBUG { 08:06:19 DEBUG "status": "success", 08:06:19 DEBUG "data": { 08:06:19 DEBUG "resultType": "vector", 08:06:19 DEBUG "result": [ 08:06:19 DEBUG { 08:06:19 DEBUG "metric": { 08:06:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG [...] 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 1686553579, 08:06:19 DEBUG "0" 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG } 08:06:19 INFO [http_cmd]: http status code OK 08:06:19 DEBUG --- status code --- 08:06:19 DEBUG http status code is 200 (expected 200) 08:06:19 DEBUG --- http response --- 08:06:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:06:19 DEBUG ----- output ----- 08:06:19 DEBUG { 08:06:19 DEBUG "status": "success", 08:06:19 DEBUG "data": { 08:06:19 DEBUG "resultType": "vector", 08:06:19 DEBUG "result": [ 08:06:19 DEBUG { 08:06:19 DEBUG "metric": { 08:06:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG [...] 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 1686553579, 08:06:19 DEBUG "14428962816" 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG } 08:06:19 INFO [http_cmd]: http status code OK 08:06:19 DEBUG --- status code --- 08:06:19 DEBUG http status code is 200 (expected 200) 08:06:19 DEBUG --- http response --- 08:06:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:06:19 DEBUG ----- output ----- 08:06:19 DEBUG { 08:06:19 DEBUG "status": "success", 08:06:19 DEBUG "data": { 08:06:19 DEBUG "resultType": "vector", 08:06:19 DEBUG "result": [ 08:06:19 DEBUG { 08:06:19 DEBUG "metric": { 08:06:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG [...] 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG "pod": "ds-cts-0" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 1686553579, 08:06:19 DEBUG "0" 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG } 08:06:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553579" 08:06:19 INFO [http_cmd]: http status code OK 08:06:19 DEBUG --- status code --- 08:06:19 DEBUG http status code is 200 (expected 200) 08:06:19 DEBUG --- http response --- 08:06:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:06:19 DEBUG ----- output ----- 08:06:19 DEBUG { 08:06:19 DEBUG "status": "success", 08:06:19 DEBUG "data": { 08:06:19 DEBUG "resultType": "vector", 08:06:19 DEBUG "result": [ 08:06:19 DEBUG { 08:06:19 DEBUG "metric": { 08:06:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG [...] 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 1686553579, 08:06:19 DEBUG "0" 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG } 08:06:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553579" 08:06:19 INFO [http_cmd]: http status code OK 08:06:19 DEBUG --- status code --- 08:06:19 DEBUG http status code is 200 (expected 200) 08:06:19 DEBUG --- http response --- 08:06:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:06:19 DEBUG ----- output ----- 08:06:19 DEBUG { 08:06:19 DEBUG "status": "success", 08:06:19 DEBUG "data": { 08:06:19 DEBUG "resultType": "vector", 08:06:19 DEBUG "result": [ 08:06:19 DEBUG { 08:06:19 DEBUG "metric": { 08:06:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG [...] 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG "pod": "ds-cts-0" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 1686553579, 08:06:19 DEBUG "0" 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG } 08:06:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553579" 08:06:19 INFO [http_cmd]: http status code OK 08:06:19 DEBUG --- status code --- 08:06:19 DEBUG http status code is 200 (expected 200) 08:06:19 DEBUG --- http response --- 08:06:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:06:19 DEBUG ----- output ----- 08:06:19 DEBUG { 08:06:19 DEBUG "status": "success", 08:06:19 DEBUG "data": { 08:06:19 DEBUG "resultType": "vector", 08:06:19 DEBUG "result": [ 08:06:19 DEBUG { 08:06:19 DEBUG "metric": { 08:06:19 DEBUG "instance": "10.142.0.100:9100" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG [...] 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG "instance": "10.142.0.99:9100" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 1686553579, 08:06:19 DEBUG "3886595.533333333" 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG } 08:06:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553579" 08:06:19 INFO [http_cmd]: http status code OK 08:06:19 DEBUG --- status code --- 08:06:19 DEBUG http status code is 200 (expected 200) 08:06:19 DEBUG --- http response --- 08:06:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:06:19 DEBUG ----- output ----- 08:06:19 DEBUG { 08:06:19 DEBUG "status": "success", 08:06:19 DEBUG "data": { 08:06:19 DEBUG "resultType": "vector", 08:06:19 DEBUG "result": [ 08:06:19 DEBUG { 08:06:19 DEBUG "metric": { 08:06:19 DEBUG "pod": "ds-cts-2" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG [...] 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG 08:06:19 DEBUG "pod": "ds-idrepo-1" 08:06:19 DEBUG }, 08:06:19 DEBUG "value": [ 08:06:19 DEBUG 1686553579, 08:06:19 DEBUG "0" 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG ] 08:06:19 DEBUG } 08:06:19 DEBUG } 08:06:52 INFO 08:06:52 INFO [loop_until]: kubectl --namespace=xlou top pods 08:06:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:06:52 INFO [loop_until]: OK (rc = 0) 08:06:52 DEBUG --- stdout --- 08:06:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 4Mi am-7849cf7bdb-bd9j2 305m 6119Mi am-7849cf7bdb-npvzp 275m 6226Mi am-7849cf7bdb-p4fns 288m 6283Mi ds-cts-0 10m 416Mi ds-cts-1 9m 451Mi ds-cts-2 10m 371Mi ds-idrepo-0 6744m 13816Mi ds-idrepo-1 2628m 13823Mi ds-idrepo-2 4916m 13757Mi end-user-ui-5c589c665b-tgm8n 1m 3Mi idm-58dc667486-ml9mc 2121m 5466Mi idm-58dc667486-vrqsf 2171m 5528Mi lodemon-75f677b97d-6b58v 6m 71Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 624m 1704Mi 08:06:52 DEBUG --- stderr --- 08:06:52 DEBUG 08:07:04 INFO 08:07:04 INFO [loop_until]: kubectl --namespace=xlou top node 08:07:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:07:04 INFO [loop_until]: OK (rc = 0) 08:07:04 DEBUG --- stdout --- 08:07:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2349m 14% 6421Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 341m 2% 7154Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 375m 2% 7049Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 2371m 14% 6418Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 350m 2% 7370Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 1462m 9% 3041Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1205Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 72m 0% 1123Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1149Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 4186m 26% 14742Mi 25% gke-xlou-cdm-ds-32e4dcb1-jr09 2624m 16% 14797Mi 25% gke-xlou-cdm-ds-32e4dcb1-vg18 68m 0% 1048Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 6822m 42% 14820Mi 25% gke-xlou-cdm-frontend-a8771548-h3n6 710m 4% 2583Mi 4% 08:07:04 DEBUG --- stderr --- 08:07:04 DEBUG 08:07:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553638" 08:07:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553638" 08:07:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553638" 08:07:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553638" 08:07:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553638" 08:07:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553638" 08:07:18 INFO [http_cmd]: http status code OK 08:07:18 DEBUG --- status code --- 08:07:18 INFO [http_cmd]: http status code OK 08:07:18 INFO [http_cmd]: http status code OK 08:07:18 INFO [http_cmd]: http status code OK 08:07:18 INFO [http_cmd]: http status code OK 08:07:18 DEBUG http status code is 200 (expected 200) 08:07:18 DEBUG --- status code --- 08:07:18 DEBUG --- status code --- 08:07:18 DEBUG --- status code --- 08:07:18 DEBUG --- status code --- 08:07:18 DEBUG --- http response --- 08:07:18 DEBUG http status code is 200 (expected 200) 08:07:18 DEBUG http status code is 200 (expected 200) 08:07:18 DEBUG http status code is 200 (expected 200) 08:07:18 DEBUG http status code is 200 (expected 200) 08:07:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:07:18 DEBUG --- http response --- 08:07:18 DEBUG --- http response --- 08:07:18 DEBUG --- http response --- 08:07:18 DEBUG --- http response --- 08:07:18 DEBUG ----- output ----- 08:07:18 INFO [http_cmd]: http status code OK 08:07:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:07:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:07:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:07:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:07:18 DEBUG { 08:07:18 DEBUG --- status code --- 08:07:18 DEBUG ----- output ----- 08:07:18 DEBUG ----- output ----- 08:07:18 DEBUG ----- output ----- 08:07:18 DEBUG ----- output ----- 08:07:18 DEBUG "status": "success", 08:07:18 DEBUG http status code is 200 (expected 200) 08:07:18 DEBUG { 08:07:18 DEBUG { 08:07:18 DEBUG { 08:07:18 DEBUG { 08:07:18 DEBUG "data": { 08:07:18 DEBUG --- http response --- 08:07:18 DEBUG "status": "success", 08:07:18 DEBUG "status": "success", 08:07:18 DEBUG "status": "success", 08:07:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553638" 08:07:18 DEBUG "status": "success", 08:07:18 DEBUG "resultType": "vector", 08:07:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:07:18 DEBUG "data": { 08:07:18 DEBUG "data": { 08:07:18 DEBUG "data": { 08:07:18 DEBUG "data": { 08:07:18 DEBUG "result": [ 08:07:18 DEBUG ----- output ----- 08:07:18 DEBUG "resultType": "vector", 08:07:18 DEBUG "resultType": "vector", 08:07:18 DEBUG "resultType": "vector", 08:07:18 DEBUG "resultType": "vector", 08:07:18 DEBUG { 08:07:18 DEBUG { 08:07:18 DEBUG "result": [ 08:07:18 DEBUG "result": [ 08:07:18 DEBUG "result": [ 08:07:18 DEBUG "result": [ 08:07:18 DEBUG "metric": { 08:07:18 DEBUG "status": "success", 08:07:18 DEBUG { 08:07:18 DEBUG { 08:07:18 DEBUG { 08:07:18 DEBUG { 08:07:18 DEBUG "pod": "ds-idrepo-2" 08:07:18 DEBUG "data": { 08:07:18 DEBUG "metric": { 08:07:18 DEBUG "metric": { 08:07:18 DEBUG "metric": { 08:07:18 DEBUG "metric": { 08:07:18 DEBUG }, 08:07:18 DEBUG "resultType": "vector", 08:07:18 DEBUG "pod": "ds-cts-2" 08:07:18 DEBUG "instance": "10.142.0.100:9100" 08:07:18 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:07:18 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:07:18 DEBUG "value": [ 08:07:18 DEBUG "result": [ 08:07:18 DEBUG }, 08:07:18 DEBUG }, 08:07:18 DEBUG }, 08:07:18 DEBUG }, 08:07:18 DEBUG 08:07:18 DEBUG { 08:07:18 DEBUG "value": [ 08:07:18 DEBUG "value": [ 08:07:18 DEBUG "value": [ 08:07:18 DEBUG "value": [ 08:07:18 DEBUG 08:07:18 DEBUG "metric": { 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG [...] 08:07:18 DEBUG }, 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG "value": [ 08:07:18 DEBUG [...] 08:07:18 DEBUG [...] 08:07:18 DEBUG [...] 08:07:18 DEBUG [...] 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG "pod": "ds-idrepo-1" 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG }, 08:07:18 DEBUG [...] 08:07:18 DEBUG "pod": "ds-idrepo-1" 08:07:18 DEBUG "instance": "10.142.0.99:9100" 08:07:18 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:07:18 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:07:18 DEBUG "value": [ 08:07:18 DEBUG 08:07:18 DEBUG }, 08:07:18 DEBUG }, 08:07:18 DEBUG }, 08:07:18 DEBUG }, 08:07:18 DEBUG 1686553638, 08:07:18 DEBUG 08:07:18 DEBUG "value": [ 08:07:18 DEBUG "value": [ 08:07:18 DEBUG "value": [ 08:07:18 DEBUG "value": [ 08:07:18 DEBUG "40548.96666666667" 08:07:18 DEBUG 08:07:18 DEBUG 1686553638, 08:07:18 DEBUG 1686553638, 08:07:18 DEBUG 1686553638, 08:07:18 DEBUG 1686553638, 08:07:18 DEBUG ] 08:07:18 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:07:18 DEBUG "1557.7666666666667" 08:07:18 DEBUG "3882936.833333333" 08:07:18 DEBUG "0" 08:07:18 DEBUG "0" 08:07:18 DEBUG } 08:07:18 DEBUG }, 08:07:18 DEBUG ] 08:07:18 DEBUG ] 08:07:18 DEBUG ] 08:07:18 DEBUG ] 08:07:18 DEBUG ] 08:07:18 DEBUG "value": [ 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 DEBUG 1686553638, 08:07:18 DEBUG ] 08:07:18 DEBUG ] 08:07:18 DEBUG ] 08:07:18 DEBUG ] 08:07:18 DEBUG } 08:07:18 DEBUG "0" 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 INFO [http_cmd]: http status code OK 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 DEBUG ] 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 DEBUG --- status code --- 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 DEBUG http status code is 200 (expected 200) 08:07:18 DEBUG ] 08:07:18 DEBUG --- http response --- 08:07:18 DEBUG } 08:07:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:07:18 DEBUG } 08:07:18 DEBUG ----- output ----- 08:07:18 DEBUG { 08:07:18 DEBUG "status": "success", 08:07:18 DEBUG "data": { 08:07:18 DEBUG "resultType": "vector", 08:07:18 DEBUG "result": [ 08:07:18 DEBUG { 08:07:18 DEBUG "metric": { 08:07:18 DEBUG "pod": "ds-idrepo-2" 08:07:18 DEBUG }, 08:07:18 DEBUG "value": [ 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG [...] 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG "pod": "ds-idrepo-1" 08:07:18 DEBUG }, 08:07:18 DEBUG "value": [ 08:07:18 DEBUG 1686553638, 08:07:18 DEBUG "6621.833333333333" 08:07:18 DEBUG ] 08:07:18 DEBUG } 08:07:18 DEBUG ] 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553638" 08:07:18 INFO [http_cmd]: http status code OK 08:07:18 DEBUG --- status code --- 08:07:18 DEBUG http status code is 200 (expected 200) 08:07:18 DEBUG --- http response --- 08:07:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:07:18 DEBUG ----- output ----- 08:07:18 DEBUG { 08:07:18 DEBUG "status": "success", 08:07:18 DEBUG "data": { 08:07:18 DEBUG "resultType": "vector", 08:07:18 DEBUG "result": [ 08:07:18 DEBUG { 08:07:18 DEBUG "metric": { 08:07:18 DEBUG "pod": "ds-cts-2" 08:07:18 DEBUG }, 08:07:18 DEBUG "value": [ 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG [...] 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG "pod": "ds-idrepo-1" 08:07:18 DEBUG }, 08:07:18 DEBUG "value": [ 08:07:18 DEBUG 1686553638, 08:07:18 DEBUG "0" 08:07:18 DEBUG ] 08:07:18 DEBUG } 08:07:18 DEBUG ] 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553638" 08:07:18 INFO [http_cmd]: http status code OK 08:07:18 DEBUG --- status code --- 08:07:18 DEBUG http status code is 200 (expected 200) 08:07:18 DEBUG --- http response --- 08:07:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:07:18 DEBUG ----- output ----- 08:07:18 DEBUG { 08:07:18 DEBUG "status": "success", 08:07:18 DEBUG "data": { 08:07:18 DEBUG "resultType": "vector", 08:07:18 DEBUG "result": [ 08:07:18 DEBUG { 08:07:18 DEBUG "metric": { 08:07:18 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:07:18 DEBUG }, 08:07:18 DEBUG "value": [ 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG [...] 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG 08:07:18 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:07:18 DEBUG }, 08:07:18 DEBUG "value": [ 08:07:18 DEBUG 1686553638, 08:07:18 DEBUG "0" 08:07:18 DEBUG ] 08:07:18 DEBUG } 08:07:18 DEBUG ] 08:07:18 DEBUG } 08:07:18 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553639" 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686553639" 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "pod": "ds-idrepo-2" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "pod": "ds-idrepo-1" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "33642.066666666666" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "0.6826937236212268" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553639" 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "instance": "10.142.0.100:9100" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "instance": "10.142.0.99:9100" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "0.0016666666666666607" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553639" 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553639" 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "instance": "10.142.0.100:9100" 08:07:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:07:19 DEBUG }, 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "instance": "10.142.0.99:9100" 08:07:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:07:19 DEBUG }, 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "0.0023666666666675457" 08:07:19 DEBUG "0" 08:07:19 DEBUG ] 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553639" 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "0.2" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686553639" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553639" 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "14556246016" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "pod": "idm-58dc667486-vrqsf" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "0" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553639" 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "0" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553639" 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "pod": "idm-58dc667486-vrqsf" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "0" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553639" 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "instance": "10.142.0.100:9100" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "instance": "10.142.0.99:9100" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "3877609.0999999996" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553639" 08:07:19 INFO [http_cmd]: http status code OK 08:07:19 DEBUG --- status code --- 08:07:19 DEBUG http status code is 200 (expected 200) 08:07:19 DEBUG --- http response --- 08:07:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:07:19 DEBUG ----- output ----- 08:07:19 DEBUG { 08:07:19 DEBUG "status": "success", 08:07:19 DEBUG "data": { 08:07:19 DEBUG "resultType": "vector", 08:07:19 DEBUG "result": [ 08:07:19 DEBUG { 08:07:19 DEBUG "metric": { 08:07:19 DEBUG "pod": "ds-cts-2" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG [...] 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG 08:07:19 DEBUG "pod": "ds-idrepo-1" 08:07:19 DEBUG }, 08:07:19 DEBUG "value": [ 08:07:19 DEBUG 1686553639, 08:07:19 DEBUG "0" 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG ] 08:07:19 DEBUG } 08:07:19 DEBUG } 08:07:52 INFO 08:07:52 INFO [loop_until]: kubectl --namespace=xlou top pods 08:07:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:07:52 INFO [loop_until]: OK (rc = 0) 08:07:52 DEBUG --- stdout --- 08:07:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 4Mi am-7849cf7bdb-bd9j2 278m 6119Mi am-7849cf7bdb-npvzp 309m 6228Mi am-7849cf7bdb-p4fns 278m 6284Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 11m 370Mi ds-idrepo-0 6878m 13733Mi ds-idrepo-1 3714m 13678Mi ds-idrepo-2 3373m 13714Mi end-user-ui-5c589c665b-tgm8n 1m 4Mi idm-58dc667486-ml9mc 2096m 5465Mi idm-58dc667486-vrqsf 2100m 5526Mi lodemon-75f677b97d-6b58v 4m 71Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 646m 1704Mi 08:07:52 DEBUG --- stderr --- 08:07:52 DEBUG 08:08:04 INFO 08:08:04 INFO [loop_until]: kubectl --namespace=xlou top node 08:08:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:08:04 INFO [loop_until]: OK (rc = 0) 08:08:04 DEBUG --- stdout --- 08:08:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2348m 14% 6418Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 378m 2% 7160Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 322m 2% 7048Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 2291m 14% 6421Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 347m 2% 7376Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 1484m 9% 3053Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1205Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 72m 0% 1119Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1152Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 3548m 22% 14683Mi 25% gke-xlou-cdm-ds-32e4dcb1-jr09 4023m 25% 14665Mi 25% gke-xlou-cdm-ds-32e4dcb1-vg18 70m 0% 1045Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 7277m 45% 14718Mi 25% gke-xlou-cdm-frontend-a8771548-h3n6 726m 4% 2583Mi 4% 08:08:04 DEBUG --- stderr --- 08:08:04 DEBUG 08:08:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553698" 08:08:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553698" 08:08:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553698" 08:08:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553698" 08:08:18 INFO [http_cmd]: http status code OK 08:08:18 DEBUG --- status code --- 08:08:18 INFO [http_cmd]: http status code OK 08:08:18 INFO [http_cmd]: http status code OK 08:08:18 DEBUG http status code is 200 (expected 200) 08:08:18 DEBUG --- status code --- 08:08:18 DEBUG --- status code --- 08:08:18 DEBUG --- http response --- 08:08:18 DEBUG http status code is 200 (expected 200) 08:08:18 DEBUG http status code is 200 (expected 200) 08:08:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:08:18 DEBUG --- http response --- 08:08:18 DEBUG --- http response --- 08:08:18 DEBUG ----- output ----- 08:08:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:08:18 DEBUG { 08:08:18 DEBUG ----- output ----- 08:08:18 DEBUG ----- output ----- 08:08:18 DEBUG "status": "success", 08:08:18 DEBUG { 08:08:18 DEBUG { 08:08:18 DEBUG "data": { 08:08:18 DEBUG "status": "success", 08:08:18 DEBUG "status": "success", 08:08:18 DEBUG "resultType": "vector", 08:08:18 DEBUG "data": { 08:08:18 DEBUG "data": { 08:08:18 DEBUG "result": [ 08:08:18 DEBUG "resultType": "vector", 08:08:18 DEBUG "resultType": "vector", 08:08:18 DEBUG { 08:08:18 DEBUG "result": [ 08:08:18 DEBUG "result": [ 08:08:18 DEBUG "metric": { 08:08:18 DEBUG { 08:08:18 DEBUG { 08:08:18 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:08:18 DEBUG "metric": { 08:08:18 DEBUG "metric": { 08:08:18 DEBUG }, 08:08:18 DEBUG "instance": "10.142.0.100:9100" 08:08:18 DEBUG "pod": "ds-idrepo-2" 08:08:18 DEBUG "value": [ 08:08:18 DEBUG }, 08:08:18 DEBUG }, 08:08:18 DEBUG 08:08:18 DEBUG "value": [ 08:08:18 DEBUG "value": [ 08:08:18 DEBUG 08:08:18 INFO [http_cmd]: http status code OK 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG --- status code --- 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG [...] 08:08:18 DEBUG http status code is 200 (expected 200) 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG --- http response --- 08:08:18 DEBUG [...] 08:08:18 DEBUG [...] 08:08:18 DEBUG 08:08:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG ----- output ----- 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:08:18 DEBUG { 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG }, 08:08:18 DEBUG "status": "success", 08:08:18 DEBUG "instance": "10.142.0.99:9100" 08:08:18 DEBUG "pod": "ds-idrepo-1" 08:08:18 DEBUG "value": [ 08:08:18 DEBUG "data": { 08:08:18 DEBUG }, 08:08:18 DEBUG }, 08:08:18 DEBUG 1686553698, 08:08:18 DEBUG "resultType": "vector", 08:08:18 DEBUG "value": [ 08:08:18 DEBUG "value": [ 08:08:18 DEBUG "0" 08:08:18 DEBUG "result": [ 08:08:18 DEBUG 1686553698, 08:08:18 DEBUG 1686553698, 08:08:18 DEBUG ] 08:08:18 DEBUG { 08:08:18 DEBUG "3903175.5666666664" 08:08:18 DEBUG "6443.066666666667" 08:08:18 DEBUG } 08:08:18 DEBUG "metric": { 08:08:18 DEBUG ] 08:08:18 DEBUG ] 08:08:18 DEBUG ] 08:08:18 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:08:18 DEBUG } 08:08:18 DEBUG } 08:08:18 DEBUG } 08:08:18 DEBUG }, 08:08:18 DEBUG ] 08:08:18 DEBUG ] 08:08:18 DEBUG } 08:08:18 DEBUG "value": [ 08:08:18 DEBUG } 08:08:18 DEBUG } 08:08:18 DEBUG 08:08:18 DEBUG } 08:08:18 DEBUG } 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG [...] 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG 08:08:18 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:08:18 DEBUG }, 08:08:18 DEBUG "value": [ 08:08:18 DEBUG 1686553698, 08:08:18 DEBUG "0" 08:08:18 DEBUG ] 08:08:18 DEBUG } 08:08:18 DEBUG ] 08:08:18 DEBUG } 08:08:18 DEBUG } 08:08:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553698" 08:08:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553698" 08:08:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553698" 08:08:18 INFO [http_cmd]: http status code OK 08:08:18 DEBUG --- status code --- 08:08:18 INFO [http_cmd]: http status code OK 08:08:18 INFO [http_cmd]: http status code OK 08:08:18 DEBUG http status code is 200 (expected 200) 08:08:18 DEBUG --- status code --- 08:08:18 DEBUG --- status code --- 08:08:18 DEBUG --- http response --- 08:08:18 DEBUG http status code is 200 (expected 200) 08:08:18 DEBUG http status code is 200 (expected 200) 08:08:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:08:18 DEBUG --- http response --- 08:08:18 DEBUG --- http response --- 08:08:18 DEBUG ----- output ----- 08:08:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:08:18 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:08:18 DEBUG { 08:08:18 DEBUG ----- output ----- 08:08:18 DEBUG ----- output ----- 08:08:18 DEBUG "status": "success", 08:08:18 DEBUG { 08:08:18 DEBUG { 08:08:18 DEBUG "data": { 08:08:18 DEBUG "status": "success", 08:08:18 DEBUG "status": "success", 08:08:18 DEBUG "resultType": "vector", 08:08:18 DEBUG "data": { 08:08:18 DEBUG "data": { 08:08:18 DEBUG "result": [ 08:08:18 DEBUG "resultType": "vector", 08:08:18 DEBUG "resultType": "vector", 08:08:18 DEBUG { 08:08:18 DEBUG "result": [ 08:08:18 DEBUG "result": [ 08:08:18 DEBUG "metric": { 08:08:18 DEBUG { 08:08:18 DEBUG { 08:08:18 DEBUG "pod": "ds-cts-2" 08:08:18 DEBUG "metric": { 08:08:18 DEBUG "metric": { 08:08:18 DEBUG }, 08:08:18 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:08:18 DEBUG "pod": "ds-idrepo-2" 08:08:19 DEBUG "value": [ 08:08:19 DEBUG }, 08:08:19 DEBUG }, 08:08:19 DEBUG 08:08:19 DEBUG "value": [ 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "pod": "ds-idrepo-1" 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG }, 08:08:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:08:19 DEBUG "pod": "ds-idrepo-1" 08:08:19 DEBUG "value": [ 08:08:19 DEBUG }, 08:08:19 DEBUG }, 08:08:19 DEBUG 1686553698, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG "value": [ 08:08:19 DEBUG "1537.7" 08:08:19 DEBUG 1686553698, 08:08:19 DEBUG 1686553698, 08:08:19 DEBUG ] 08:08:19 DEBUG "0" 08:08:19 DEBUG "40516.63333333333" 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG ] 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "pod": "ds-cts-2" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "pod": "ds-idrepo-1" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "0" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "0" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553699" 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "pod": "ds-idrepo-2" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "pod": "ds-idrepo-1" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "33537.6" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "0.6754061015204689" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "instance": "10.142.0.100:9100" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "instance": "10.142.0.99:9100" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "0.0010000000000000083" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553699" 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:19 DEBUG { 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG { 08:08:19 DEBUG "data": { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "result": [ 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG { 08:08:19 DEBUG "result": [ 08:08:19 DEBUG "metric": { 08:08:19 DEBUG { 08:08:19 DEBUG "instance": "10.142.0.100:9100" 08:08:19 DEBUG "metric": { 08:08:19 DEBUG }, 08:08:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:08:19 DEBUG "value": [ 08:08:19 DEBUG }, 08:08:19 DEBUG 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "instance": "10.142.0.99:9100" 08:08:19 DEBUG 08:08:19 DEBUG }, 08:08:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:08:19 DEBUG "value": [ 08:08:19 DEBUG }, 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG "0.002333333333333106" 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG ] 08:08:19 DEBUG "0" 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "0.2" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686553699" 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "14462107648" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "0.2989736310762712" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "0" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "32529.028574394644" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553699" 08:08:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553699" 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "pod": "ds-cts-2" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "pod": "ds-idrepo-1" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "0" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:19 INFO [http_cmd]: http status code OK 08:08:19 DEBUG --- status code --- 08:08:19 DEBUG http status code is 200 (expected 200) 08:08:19 DEBUG --- http response --- 08:08:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:08:19 DEBUG ----- output ----- 08:08:19 DEBUG { 08:08:19 DEBUG "status": "success", 08:08:19 DEBUG "data": { 08:08:19 DEBUG "resultType": "vector", 08:08:19 DEBUG "result": [ 08:08:19 DEBUG { 08:08:19 DEBUG "metric": { 08:08:19 DEBUG "instance": "10.142.0.100:9100" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG [...] 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG 08:08:19 DEBUG "instance": "10.142.0.99:9100" 08:08:19 DEBUG }, 08:08:19 DEBUG "value": [ 08:08:19 DEBUG 1686553699, 08:08:19 DEBUG "3899145.5" 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG ] 08:08:19 DEBUG } 08:08:19 DEBUG } 08:08:52 INFO 08:08:52 INFO [loop_until]: kubectl --namespace=xlou top pods 08:08:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:08:52 INFO [loop_until]: OK (rc = 0) 08:08:52 DEBUG --- stdout --- 08:08:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 4Mi am-7849cf7bdb-bd9j2 325m 6120Mi am-7849cf7bdb-npvzp 280m 6228Mi am-7849cf7bdb-p4fns 283m 6284Mi ds-cts-0 10m 417Mi ds-cts-1 8m 451Mi ds-cts-2 10m 371Mi ds-idrepo-0 6780m 13718Mi ds-idrepo-1 3386m 13678Mi ds-idrepo-2 3854m 13780Mi end-user-ui-5c589c665b-tgm8n 1m 3Mi idm-58dc667486-ml9mc 2089m 5465Mi idm-58dc667486-vrqsf 2175m 5526Mi lodemon-75f677b97d-6b58v 5m 70Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 663m 1704Mi 08:08:52 DEBUG --- stderr --- 08:08:52 DEBUG 08:09:04 INFO 08:09:04 INFO [loop_until]: kubectl --namespace=xlou top node 08:09:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:09:04 INFO [loop_until]: OK (rc = 0) 08:09:04 DEBUG --- stdout --- 08:09:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2345m 14% 6419Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 342m 2% 7161Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 385m 2% 7049Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 2370m 14% 6420Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 333m 2% 7373Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 1476m 9% 3056Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1207Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 69m 0% 1118Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1152Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 3501m 22% 14775Mi 25% gke-xlou-cdm-ds-32e4dcb1-jr09 3583m 22% 14680Mi 25% gke-xlou-cdm-ds-32e4dcb1-vg18 67m 0% 1045Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 7440m 46% 14717Mi 25% gke-xlou-cdm-frontend-a8771548-h3n6 743m 4% 2580Mi 4% 08:09:04 DEBUG --- stderr --- 08:09:04 DEBUG 08:09:18 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553758" 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553759" 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "resultType": "vector", 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG "result": [ 08:09:19 DEBUG "result": [ 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG { 08:09:19 DEBUG { 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG "pod": "ds-idrepo-2" 08:09:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:09:19 DEBUG }, 08:09:19 DEBUG }, 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG "value": [ 08:09:19 DEBUG "value": [ 08:09:19 DEBUG { 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "data": { 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG [...] 08:09:19 DEBUG [...] 08:09:19 DEBUG "result": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG { 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "metric": { 08:09:19 DEBUG 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG "instance": "10.142.0.100:9100" 08:09:19 DEBUG "pod": "ds-idrepo-1" 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG "pod": "ds-cts-1" 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG }, 08:09:19 DEBUG }, 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG }, 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG "value": [ 08:09:19 DEBUG "value": [ 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG "value": [ 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG 08:09:19 DEBUG 1686553758, 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:09:19 DEBUG 08:09:19 DEBUG "6616.233333333334" 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG "0" 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG 08:09:19 DEBUG ] 08:09:19 DEBUG { 08:09:19 DEBUG ] 08:09:19 DEBUG { 08:09:19 DEBUG [...] 08:09:19 DEBUG } 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG } 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG 08:09:19 DEBUG ] 08:09:19 DEBUG "data": { 08:09:19 DEBUG ] 08:09:19 DEBUG "data": { 08:09:19 DEBUG 08:09:19 DEBUG } 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG } 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG 08:09:19 DEBUG } 08:09:19 DEBUG "result": [ 08:09:19 DEBUG } 08:09:19 DEBUG "result": [ 08:09:19 DEBUG "instance": "10.142.0.99:9100" 08:09:19 DEBUG { 08:09:19 DEBUG { 08:09:19 DEBUG }, 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "value": [ 08:09:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:09:19 DEBUG "pod": "ds-idrepo-2" 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG }, 08:09:19 DEBUG }, 08:09:19 DEBUG "3969812.3" 08:09:19 DEBUG "value": [ 08:09:19 DEBUG "value": [ 08:09:19 DEBUG ] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG } 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG ] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG } 08:09:19 DEBUG [...] 08:09:19 DEBUG [...] 08:09:19 DEBUG } 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:09:19 DEBUG "pod": "ds-idrepo-1" 08:09:19 DEBUG }, 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0" 08:09:19 DEBUG "40986.26666666666" 08:09:19 DEBUG ] 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "pod": "ds-cts-2" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "ds-idrepo-1" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "pod": "ds-cts-2" 08:09:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:09:19 DEBUG }, 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "ds-idrepo-1" 08:09:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:09:19 DEBUG }, 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "1806.2" 08:09:19 DEBUG "0" 08:09:19 DEBUG ] 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "pod": "ds-idrepo-2" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "ds-idrepo-1" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "33962.46666666667" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0.7071905159399471" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "instance": "10.142.0.100:9100" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "instance": "10.142.0.99:9100" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0.0013333333333333049" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553759" 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "instance": "10.142.0.100:9100" 08:09:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:09:19 DEBUG }, 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG "value": [ 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686553759" 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "instance": "10.142.0.99:9100" 08:09:19 DEBUG "pod": "ds-cts-1" 08:09:19 DEBUG }, 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0.0024333333333345307" 08:09:19 DEBUG "0" 08:09:19 DEBUG ] 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "14572883968" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0.2" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "pod": "ds-cts-2" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "pod": "ds-idrepo-1" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "0" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553759" 08:09:19 INFO [http_cmd]: http status code OK 08:09:19 DEBUG --- status code --- 08:09:19 DEBUG http status code is 200 (expected 200) 08:09:19 DEBUG --- http response --- 08:09:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:09:19 DEBUG ----- output ----- 08:09:19 DEBUG { 08:09:19 DEBUG "status": "success", 08:09:19 DEBUG "data": { 08:09:19 DEBUG "resultType": "vector", 08:09:19 DEBUG "result": [ 08:09:19 DEBUG { 08:09:19 DEBUG "metric": { 08:09:19 DEBUG "instance": "10.142.0.100:9100" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG [...] 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG 08:09:19 DEBUG "instance": "10.142.0.99:9100" 08:09:19 DEBUG }, 08:09:19 DEBUG "value": [ 08:09:19 DEBUG 1686553759, 08:09:19 DEBUG "3964117.666666667" 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG ] 08:09:19 DEBUG } 08:09:19 DEBUG } 08:09:52 INFO 08:09:52 INFO [loop_until]: kubectl --namespace=xlou top pods 08:09:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:09:52 INFO [loop_until]: OK (rc = 0) 08:09:52 DEBUG --- stdout --- 08:09:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 4Mi am-7849cf7bdb-bd9j2 285m 6120Mi am-7849cf7bdb-npvzp 279m 6228Mi am-7849cf7bdb-p4fns 329m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 5739m 13825Mi ds-idrepo-1 4249m 13824Mi ds-idrepo-2 4026m 13809Mi end-user-ui-5c589c665b-tgm8n 1m 3Mi idm-58dc667486-ml9mc 2125m 5464Mi idm-58dc667486-vrqsf 2186m 5526Mi lodemon-75f677b97d-6b58v 6m 71Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 645m 1704Mi 08:09:52 DEBUG --- stderr --- 08:09:52 DEBUG 08:10:04 INFO 08:10:04 INFO [loop_until]: kubectl --namespace=xlou top node 08:10:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:10:05 INFO [loop_until]: OK (rc = 0) 08:10:05 DEBUG --- stdout --- 08:10:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2347m 14% 6419Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 335m 2% 7158Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 341m 2% 7053Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 2401m 15% 6418Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 392m 2% 7375Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 1522m 9% 3045Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 86m 0% 1219Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 69m 0% 1120Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1151Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 3901m 24% 14660Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 3582m 22% 14795Mi 25% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1050Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 5168m 32% 14822Mi 25% gke-xlou-cdm-frontend-a8771548-h3n6 722m 4% 2579Mi 4% 08:10:05 DEBUG --- stderr --- 08:10:05 DEBUG 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "ds-idrepo-2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "ds-idrepo-1" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "6358.533333333333" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG }, 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG "value": [ 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG "0" 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:10:19 DEBUG ] 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG } 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG ] 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG } 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG } 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG { 08:10:19 DEBUG "pod": "ds-idrepo-2" 08:10:19 DEBUG "metric": { 08:10:19 DEBUG }, 08:10:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:10:19 DEBUG "value": [ 08:10:19 DEBUG }, 08:10:19 DEBUG 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "ds-idrepo-1" 08:10:19 DEBUG 08:10:19 DEBUG }, 08:10:19 DEBUG "pod": "ds-cts-0" 08:10:19 DEBUG "value": [ 08:10:19 DEBUG }, 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG "40853.1" 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG ] 08:10:19 DEBUG "0" 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "instance": "10.142.0.100:9100" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "instance": "10.142.0.99:9100" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "3933123.0666666664" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "ds-cts-2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "ds-idrepo-1" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "1283.2666666666667" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "ds-cts-2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "ds-idrepo-1" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553819" 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "instance": "10.142.0.100:9100" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "instance": "10.142.0.99:9100" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0.0013333333333333493" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "ds-idrepo-2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "ds-idrepo-1" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "33796.26666666666" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0.6355763678138286" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553819" 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "instance": "10.142.0.100:9100" 08:10:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:10:19 DEBUG }, 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "instance": "10.142.0.99:9100" 08:10:19 DEBUG "pod": "ds-cts-0" 08:10:19 DEBUG }, 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0.002400000000000091" 08:10:19 DEBUG "0" 08:10:19 DEBUG ] 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "14472126464" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0.2" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "ds-cts-0" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0.009640394882301388" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "ds-cts-0" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "316.6479842294461" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "pod": "ds-cts-2" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "pod": "ds-idrepo-1" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "0" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553819" 08:10:19 INFO [http_cmd]: http status code OK 08:10:19 DEBUG --- status code --- 08:10:19 DEBUG http status code is 200 (expected 200) 08:10:19 DEBUG --- http response --- 08:10:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:10:19 DEBUG ----- output ----- 08:10:19 DEBUG { 08:10:19 DEBUG "status": "success", 08:10:19 DEBUG "data": { 08:10:19 DEBUG "resultType": "vector", 08:10:19 DEBUG "result": [ 08:10:19 DEBUG { 08:10:19 DEBUG "metric": { 08:10:19 DEBUG "instance": "10.142.0.100:9100" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG [...] 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG 08:10:19 DEBUG "instance": "10.142.0.99:9100" 08:10:19 DEBUG }, 08:10:19 DEBUG "value": [ 08:10:19 DEBUG 1686553819, 08:10:19 DEBUG "3928963.7" 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG ] 08:10:19 DEBUG } 08:10:19 DEBUG } 08:10:52 INFO 08:10:52 INFO [loop_until]: kubectl --namespace=xlou top pods 08:10:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:10:52 INFO [loop_until]: OK (rc = 0) 08:10:52 DEBUG --- stdout --- 08:10:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 4Mi am-7849cf7bdb-bd9j2 273m 6120Mi am-7849cf7bdb-npvzp 274m 6229Mi am-7849cf7bdb-p4fns 259m 6285Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 10m 371Mi ds-idrepo-0 6159m 13807Mi ds-idrepo-1 4032m 13637Mi ds-idrepo-2 3707m 13727Mi end-user-ui-5c589c665b-tgm8n 1m 3Mi idm-58dc667486-ml9mc 2130m 5465Mi idm-58dc667486-vrqsf 2078m 5527Mi lodemon-75f677b97d-6b58v 5m 71Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 651m 1705Mi 08:10:52 DEBUG --- stderr --- 08:10:52 DEBUG 08:11:05 INFO 08:11:05 INFO [loop_until]: kubectl --namespace=xlou top node 08:11:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:11:05 INFO [loop_until]: OK (rc = 0) 08:11:05 DEBUG --- stdout --- 08:11:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2360m 14% 6418Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 339m 2% 7160Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 336m 2% 7049Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 2387m 15% 6412Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 334m 2% 7377Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 1451m 9% 3049Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1209Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1123Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1151Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 3763m 23% 14733Mi 25% gke-xlou-cdm-ds-32e4dcb1-jr09 4322m 27% 14614Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 66m 0% 1049Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 6966m 43% 14698Mi 25% gke-xlou-cdm-frontend-a8771548-h3n6 717m 4% 2581Mi 4% 08:11:05 DEBUG --- stderr --- 08:11:05 DEBUG 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553879" 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG { 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "status": "success", 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "data": { 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG "result": [ 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG { 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "instance": "10.142.0.100:9100" 08:11:19 DEBUG "pod": "ds-idrepo-2" 08:11:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:11:19 DEBUG }, 08:11:19 DEBUG }, 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG [...] 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:11:19 DEBUG 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG 08:11:19 DEBUG "instance": "10.142.0.99:9100" 08:11:19 DEBUG { 08:11:19 DEBUG "pod": "ds-idrepo-1" 08:11:19 DEBUG { 08:11:19 DEBUG "pod": "ds-cts-0" 08:11:19 DEBUG }, 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG }, 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "data": { 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "data": { 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "3855244.0999999996" 08:11:19 DEBUG "result": [ 08:11:19 DEBUG "40898.333333333336" 08:11:19 DEBUG "result": [ 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG "0" 08:11:19 DEBUG ] 08:11:19 DEBUG { 08:11:19 DEBUG ] 08:11:19 DEBUG { 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG "metric": { 08:11:19 DEBUG } 08:11:19 DEBUG "metric": { 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG "pod": "ds-idrepo-2" 08:11:19 DEBUG ] 08:11:19 DEBUG "pod": "ds-cts-2" 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG }, 08:11:19 DEBUG } 08:11:19 DEBUG }, 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 DEBUG "value": [ 08:11:19 DEBUG } 08:11:19 DEBUG "value": [ 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG } 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG { 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "data": { 08:11:19 DEBUG [...] 08:11:19 DEBUG [...] 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "result": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG { 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "pod": "ds-idrepo-1" 08:11:19 DEBUG "pod": "ds-idrepo-1" 08:11:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:11:19 DEBUG }, 08:11:19 DEBUG }, 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG 08:11:19 DEBUG "6291" 08:11:19 DEBUG "1157.5333333333333" 08:11:19 DEBUG 08:11:19 DEBUG ] 08:11:19 DEBUG ] 08:11:19 DEBUG 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 DEBUG [...] 08:11:19 DEBUG ] 08:11:19 DEBUG ] 08:11:19 DEBUG 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 DEBUG 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 DEBUG 08:11:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "0" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "pod": "ds-cts-2" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "pod": "ds-idrepo-1" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "0" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:11:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:11:19 DEBUG }, 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:11:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:11:19 DEBUG }, 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "0" 08:11:19 DEBUG "0" 08:11:19 DEBUG ] 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686553879" 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553879" 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG { 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG "status": "success", 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG "data": { 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG "result": [ 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG { 08:11:19 DEBUG { 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "metric": { 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG "data": { 08:11:19 DEBUG "pod": "ds-idrepo-2" 08:11:19 DEBUG { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG }, 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "data": { 08:11:19 DEBUG { 08:11:19 DEBUG 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "metric": { 08:11:19 DEBUG 08:11:19 DEBUG "result": [ 08:11:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:11:19 DEBUG 08:11:19 DEBUG { 08:11:19 DEBUG }, 08:11:19 DEBUG [...] 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG "instance": "10.142.0.100:9100" 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG }, 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG "pod": "ds-idrepo-1" 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG }, 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG "33809.7" 08:11:19 DEBUG 08:11:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:11:19 DEBUG ] 08:11:19 DEBUG 08:11:19 DEBUG }, 08:11:19 DEBUG } 08:11:19 DEBUG 08:11:19 DEBUG "value": [ 08:11:19 DEBUG ] 08:11:19 DEBUG "instance": "10.142.0.99:9100" 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG } 08:11:19 DEBUG }, 08:11:19 DEBUG "0.7050009406490426" 08:11:19 DEBUG } 08:11:19 DEBUG "value": [ 08:11:19 DEBUG ] 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG } 08:11:19 DEBUG "0.0010000000000000009" 08:11:19 DEBUG ] 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "pod": "ds-cts-0" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "0" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553879" 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG "result": [ 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG { 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG "metric": { 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:11:19 DEBUG "instance": "10.142.0.100:9100" 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG }, 08:11:19 DEBUG { 08:11:19 DEBUG "value": [ 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG 08:11:19 DEBUG "data": { 08:11:19 DEBUG 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG 08:11:19 DEBUG "result": [ 08:11:19 DEBUG [...] 08:11:19 DEBUG { 08:11:19 DEBUG 08:11:19 DEBUG "metric": { 08:11:19 DEBUG 08:11:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:11:19 DEBUG 08:11:19 DEBUG }, 08:11:19 DEBUG "instance": "10.142.0.99:9100" 08:11:19 DEBUG "value": [ 08:11:19 DEBUG }, 08:11:19 DEBUG 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG 08:11:19 DEBUG "0.0023333333333350006" 08:11:19 DEBUG [...] 08:11:19 DEBUG ] 08:11:19 DEBUG 08:11:19 DEBUG } 08:11:19 DEBUG 08:11:19 DEBUG ] 08:11:19 DEBUG 08:11:19 DEBUG } 08:11:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:11:19 DEBUG } 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "14534328320" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "0.2" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "0" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "0" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "0" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "pod": "ds-cts-2" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "pod": "ds-idrepo-1" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "0" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553879" 08:11:19 INFO [http_cmd]: http status code OK 08:11:19 DEBUG --- status code --- 08:11:19 DEBUG http status code is 200 (expected 200) 08:11:19 DEBUG --- http response --- 08:11:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:11:19 DEBUG ----- output ----- 08:11:19 DEBUG { 08:11:19 DEBUG "status": "success", 08:11:19 DEBUG "data": { 08:11:19 DEBUG "resultType": "vector", 08:11:19 DEBUG "result": [ 08:11:19 DEBUG { 08:11:19 DEBUG "metric": { 08:11:19 DEBUG "instance": "10.142.0.100:9100" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG [...] 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG 08:11:19 DEBUG "instance": "10.142.0.99:9100" 08:11:19 DEBUG }, 08:11:19 DEBUG "value": [ 08:11:19 DEBUG 1686553879, 08:11:19 DEBUG "3849279.2666666666" 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG ] 08:11:19 DEBUG } 08:11:19 DEBUG } 08:11:52 INFO 08:11:52 INFO [loop_until]: kubectl --namespace=xlou top pods 08:11:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:11:52 INFO [loop_until]: OK (rc = 0) 08:11:52 DEBUG --- stdout --- 08:11:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 4Mi am-7849cf7bdb-bd9j2 259m 6120Mi am-7849cf7bdb-npvzp 271m 6229Mi am-7849cf7bdb-p4fns 317m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 9m 371Mi ds-idrepo-0 6789m 13822Mi ds-idrepo-1 4442m 13573Mi ds-idrepo-2 3630m 13705Mi end-user-ui-5c589c665b-tgm8n 1m 3Mi idm-58dc667486-ml9mc 2159m 5466Mi idm-58dc667486-vrqsf 2106m 5526Mi lodemon-75f677b97d-6b58v 8m 71Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 644m 1705Mi 08:11:52 DEBUG --- stderr --- 08:11:52 DEBUG 08:12:05 INFO 08:12:05 INFO [loop_until]: kubectl --namespace=xlou top node 08:12:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:12:05 INFO [loop_until]: OK (rc = 0) 08:12:05 DEBUG --- stdout --- 08:12:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2388m 15% 6418Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 336m 2% 7162Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 333m 2% 7051Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 2304m 14% 6415Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 337m 2% 7377Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 1523m 9% 3053Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1212Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 71m 0% 1120Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1152Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 4124m 25% 14692Mi 25% gke-xlou-cdm-ds-32e4dcb1-jr09 4269m 26% 14564Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 66m 0% 1046Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 7657m 48% 14555Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 720m 4% 2581Mi 4% 08:12:05 DEBUG --- stderr --- 08:12:05 DEBUG 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553939" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG { 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "data": { 08:12:19 DEBUG { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "result": [ 08:12:19 DEBUG "data": { 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "ds-idrepo-2" 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "result": [ 08:12:19 DEBUG "instance": "10.142.0.100:9100" 08:12:19 DEBUG }, 08:12:19 DEBUG "pod": "ds-idrepo-2" 08:12:19 DEBUG }, 08:12:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:12:19 DEBUG { 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG }, 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG "value": [ 08:12:19 DEBUG "pod": "ds-cts-2" 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG }, 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:12:19 DEBUG 08:12:19 DEBUG "pod": "ds-idrepo-1" 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "instance": "10.142.0.99:9100" 08:12:19 DEBUG }, 08:12:19 DEBUG "pod": "ds-idrepo-1" 08:12:19 DEBUG }, 08:12:19 DEBUG "pod": "ds-cts-0" 08:12:19 DEBUG 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG }, 08:12:19 DEBUG 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG "pod": "ds-idrepo-1" 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "0" 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "41192.3" 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG }, 08:12:19 DEBUG "3923166" 08:12:19 DEBUG ] 08:12:19 DEBUG "6461.033333333333" 08:12:19 DEBUG ] 08:12:19 DEBUG "0" 08:12:19 DEBUG "value": [ 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG "1330.6333333333332" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "ds-cts-2" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "pod": "ds-idrepo-1" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "0" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "0" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "0" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553939" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "ds-idrepo-2" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG 08:12:19 DEBUG --- status code --- 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG [...] 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:12:19 DEBUG 08:12:19 DEBUG { 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG { 08:12:19 DEBUG "pod": "ds-idrepo-1" 08:12:19 DEBUG "data": { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG }, 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "value": [ 08:12:19 DEBUG "result": [ 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG { 08:12:19 DEBUG "result": [ 08:12:19 DEBUG "34068.066666666666" 08:12:19 DEBUG "metric": { 08:12:19 DEBUG { 08:12:19 DEBUG ] 08:12:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:12:19 DEBUG "metric": { 08:12:19 DEBUG } 08:12:19 DEBUG }, 08:12:19 DEBUG "instance": "10.142.0.100:9100" 08:12:19 DEBUG ] 08:12:19 DEBUG "value": [ 08:12:19 DEBUG }, 08:12:19 DEBUG } 08:12:19 DEBUG 08:12:19 DEBUG "value": [ 08:12:19 DEBUG } 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:12:19 DEBUG 08:12:19 DEBUG }, 08:12:19 DEBUG "instance": "10.142.0.99:9100" 08:12:19 DEBUG "value": [ 08:12:19 DEBUG }, 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG "0.6659562825507862" 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG ] 08:12:19 DEBUG "0.0009999999999999194" 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "0.2" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG { 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG "data": { 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "data": { 08:12:19 DEBUG "instance": "10.142.0.100:9100" 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG }, 08:12:19 DEBUG "result": [ 08:12:19 DEBUG "value": [ 08:12:19 DEBUG { 08:12:19 DEBUG 08:12:19 DEBUG "metric": { 08:12:19 DEBUG 08:12:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:12:19 DEBUG 08:12:19 DEBUG }, 08:12:19 DEBUG [...] 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "instance": "10.142.0.99:9100" 08:12:19 DEBUG [...] 08:12:19 DEBUG }, 08:12:19 DEBUG 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG 08:12:19 DEBUG "0.0021333333333340455" 08:12:19 DEBUG "pod": "ds-cts-0" 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 DEBUG ] 08:12:19 DEBUG }, 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686553939" 08:12:19 DEBUG "value": [ 08:12:19 DEBUG ] 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG } 08:12:19 DEBUG "0" 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "pod": "ds-cts-0" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "0.009857185736949723" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "14449197056" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "0" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "pod": "ds-cts-0" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "1130.0089661355955" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "pod": "ds-cts-2" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "pod": "ds-idrepo-1" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "0" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553939" 08:12:19 INFO [http_cmd]: http status code OK 08:12:19 DEBUG --- status code --- 08:12:19 DEBUG http status code is 200 (expected 200) 08:12:19 DEBUG --- http response --- 08:12:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:12:19 DEBUG ----- output ----- 08:12:19 DEBUG { 08:12:19 DEBUG "status": "success", 08:12:19 DEBUG "data": { 08:12:19 DEBUG "resultType": "vector", 08:12:19 DEBUG "result": [ 08:12:19 DEBUG { 08:12:19 DEBUG "metric": { 08:12:19 DEBUG "instance": "10.142.0.100:9100" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG [...] 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG 08:12:19 DEBUG "instance": "10.142.0.99:9100" 08:12:19 DEBUG }, 08:12:19 DEBUG "value": [ 08:12:19 DEBUG 1686553939, 08:12:19 DEBUG "3917400.933333333" 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG ] 08:12:19 DEBUG } 08:12:19 DEBUG } 08:12:52 INFO 08:12:52 INFO [loop_until]: kubectl --namespace=xlou top pods 08:12:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:12:52 INFO [loop_until]: OK (rc = 0) 08:12:52 DEBUG --- stdout --- 08:12:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 4Mi am-7849cf7bdb-bd9j2 107m 6120Mi am-7849cf7bdb-npvzp 102m 6231Mi am-7849cf7bdb-p4fns 140m 6286Mi ds-cts-0 9m 417Mi ds-cts-1 8m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 3771m 13630Mi ds-idrepo-1 1224m 13744Mi ds-idrepo-2 1843m 13670Mi end-user-ui-5c589c665b-tgm8n 1m 3Mi idm-58dc667486-ml9mc 718m 5464Mi idm-58dc667486-vrqsf 429m 5525Mi lodemon-75f677b97d-6b58v 7m 71Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 373m 1705Mi 08:12:52 DEBUG --- stderr --- 08:12:52 DEBUG 08:13:05 INFO 08:13:05 INFO [loop_until]: kubectl --namespace=xlou top node 08:13:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:13:05 INFO [loop_until]: OK (rc = 0) 08:13:05 DEBUG --- stdout --- 08:13:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 574m 3% 6419Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 110m 0% 7164Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 133m 0% 7052Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 552m 3% 6417Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 184m 1% 7376Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 751m 4% 3057Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1209Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 72m 0% 1119Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1149Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 1173m 7% 14638Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 1315m 8% 14718Mi 25% gke-xlou-cdm-ds-32e4dcb1-vg18 69m 0% 1047Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 2610m 16% 14620Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 363m 2% 2585Mi 4% 08:13:05 DEBUG --- stderr --- 08:13:05 DEBUG 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "ds-idrepo-2" 08:13:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:13:19 DEBUG }, 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "ds-idrepo-1" 08:13:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:13:19 DEBUG }, 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "40158.1" 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "ds-cts-2" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "ds-idrepo-1" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "ds-idrepo-2" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "ds-idrepo-1" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "5426.866666666667" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "ds-cts-2" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "ds-idrepo-1" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "instance": "10.142.0.100:9100" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "instance": "10.142.0.99:9100" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "12237.266666666666" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "ds-idrepo-2" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "ds-idrepo-1" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "33033.96666666667" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:13:19 DEBUG "instance": "10.142.0.100:9100" 08:13:19 DEBUG }, 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:13:19 DEBUG "instance": "10.142.0.99:9100" 08:13:19 DEBUG }, 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0.11308124226405798" 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0.2" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0.005792136342717984" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:13:19 DEBUG "instance": "10.142.0.100:9100" 08:13:19 DEBUG }, 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:13:19 DEBUG "instance": "10.142.0.99:9100" 08:13:19 DEBUG }, 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0" 08:13:19 DEBUG "0.0010000000000009853" 08:13:19 DEBUG ] 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "14414106624" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "pod": "ds-cts-2" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "pod": "ds-idrepo-1" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "0" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686553999" 08:13:19 INFO [http_cmd]: http status code OK 08:13:19 DEBUG --- status code --- 08:13:19 DEBUG http status code is 200 (expected 200) 08:13:19 DEBUG --- http response --- 08:13:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:19 DEBUG ----- output ----- 08:13:19 DEBUG { 08:13:19 DEBUG "status": "success", 08:13:19 DEBUG "data": { 08:13:19 DEBUG "resultType": "vector", 08:13:19 DEBUG "result": [ 08:13:19 DEBUG { 08:13:19 DEBUG "metric": { 08:13:19 DEBUG "instance": "10.142.0.100:9100" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG [...] 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG 08:13:19 DEBUG "instance": "10.142.0.99:9100" 08:13:19 DEBUG }, 08:13:19 DEBUG "value": [ 08:13:19 DEBUG 1686553999, 08:13:19 DEBUG "7308.633333333333" 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG ] 08:13:19 DEBUG } 08:13:19 DEBUG } 08:13:52 INFO 08:13:52 INFO [loop_until]: kubectl --namespace=xlou top pods 08:13:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:13:52 INFO [loop_until]: OK (rc = 0) 08:13:52 DEBUG --- stdout --- 08:13:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 4Mi am-7849cf7bdb-bd9j2 9m 6120Mi am-7849cf7bdb-npvzp 6m 6230Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 9m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 18m 13584Mi ds-idrepo-1 344m 13596Mi ds-idrepo-2 18m 13567Mi end-user-ui-5c589c665b-tgm8n 1m 3Mi idm-58dc667486-ml9mc 9m 5461Mi idm-58dc667486-vrqsf 6m 5522Mi lodemon-75f677b97d-6b58v 5m 71Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1m 527Mi 08:13:52 DEBUG --- stderr --- 08:13:52 DEBUG 08:14:05 INFO 08:14:05 INFO [loop_until]: kubectl --namespace=xlou top node 08:14:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:14:05 INFO [loop_until]: OK (rc = 0) 08:14:05 DEBUG --- stdout --- 08:14:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 68m 0% 6432Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 7158Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 7051Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 71m 0% 6419Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 66m 0% 7379Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 207m 1% 3052Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 76m 0% 1210Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1119Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1148Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 77m 0% 14531Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 274m 1% 14575Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 67m 0% 1047Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14578Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 64m 0% 1412Mi 2% 08:14:05 DEBUG --- stderr --- 08:14:05 DEBUG 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "ds-cts-2" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "ds-idrepo-1" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "data": { 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "ds-cts-2" 08:14:19 DEBUG "pod": "ds-idrepo-2" 08:14:19 DEBUG }, 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "ds-idrepo-1" 08:14:19 DEBUG "pod": "ds-idrepo-1" 08:14:19 DEBUG }, 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG "40158.1" 08:14:19 DEBUG ] 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "ds-cts-0" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "ds-idrepo-2" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "ds-idrepo-1" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "5426.866666666667" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "instance": "10.142.0.100:9100" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "instance": "10.142.0.99:9100" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "10776.666666666666" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "ds-idrepo-2" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "ds-idrepo-1" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "33033.96666666667" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "instance": "10.142.0.100:9100" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "instance": "10.142.0.99:9100" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "ds-cts-0" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0.024188436209250933" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0.2" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "ds-cts-0" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0.009837796288007378" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "instance": "10.142.0.100:9100" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "instance": "10.142.0.99:9100" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0.0001666666666665151" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "14307733504" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "ds-cts-0" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "13087937.934532868" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "pod": "ds-cts-2" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "pod": "ds-idrepo-1" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "0" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554059" 08:14:19 INFO [http_cmd]: http status code OK 08:14:19 DEBUG --- status code --- 08:14:19 DEBUG http status code is 200 (expected 200) 08:14:19 DEBUG --- http response --- 08:14:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:19 DEBUG ----- output ----- 08:14:19 DEBUG { 08:14:19 DEBUG "status": "success", 08:14:19 DEBUG "data": { 08:14:19 DEBUG "resultType": "vector", 08:14:19 DEBUG "result": [ 08:14:19 DEBUG { 08:14:19 DEBUG "metric": { 08:14:19 DEBUG "instance": "10.142.0.100:9100" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG [...] 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG 08:14:19 DEBUG "instance": "10.142.0.99:9100" 08:14:19 DEBUG }, 08:14:19 DEBUG "value": [ 08:14:19 DEBUG 1686554059, 08:14:19 DEBUG "4464.1" 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG ] 08:14:19 DEBUG } 08:14:19 DEBUG } 08:14:53 INFO 08:14:53 INFO [loop_until]: kubectl --namespace=xlou top pods 08:14:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:14:53 INFO [loop_until]: OK (rc = 0) 08:14:53 DEBUG --- stdout --- 08:14:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 14m 6125Mi am-7849cf7bdb-npvzp 12m 6233Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 79m 416Mi ds-cts-1 9m 451Mi ds-cts-2 60m 445Mi ds-idrepo-0 151m 13584Mi ds-idrepo-1 116m 13624Mi ds-idrepo-2 18m 13567Mi end-user-ui-5c589c665b-tgm8n 1m 3Mi idm-58dc667486-ml9mc 8m 5461Mi idm-58dc667486-vrqsf 5m 5522Mi lodemon-75f677b97d-6b58v 6m 71Mi login-ui-55f8bc5458-nxdzc 1m 4Mi overseer-0-5b849496f7-qmtsz 192m 806Mi 08:14:53 DEBUG --- stderr --- 08:14:53 DEBUG 08:15:05 INFO 08:15:05 INFO [loop_until]: kubectl --namespace=xlou top node 08:15:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:15:05 INFO [loop_until]: OK (rc = 0) 08:15:05 DEBUG --- stdout --- 08:15:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 74m 0% 6432Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 95m 0% 7160Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 87m 0% 7057Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 86m 0% 6414Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 95m 0% 7381Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 246m 1% 3050Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 100m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 154m 0% 1121Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1152Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 311m 1% 14533Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 229m 1% 14521Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 71m 0% 1047Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 267m 1% 14583Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 557m 3% 1758Mi 2% 08:15:05 DEBUG --- stderr --- 08:15:05 DEBUG 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG { 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG { 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG "data": { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "data": { 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:15:19 DEBUG "result": [ 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "pod": "ds-idrepo-2" 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "data": { 08:15:19 DEBUG }, 08:15:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "value": [ 08:15:19 DEBUG }, 08:15:19 DEBUG "result": [ 08:15:19 DEBUG 08:15:19 DEBUG "value": [ 08:15:19 DEBUG { 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "metric": { 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "ds-cts-2" 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG }, 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "ds-idrepo-1" 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG }, 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:15:19 DEBUG [...] 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG "value": [ 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG }, 08:15:19 DEBUG 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG "0" 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:19 DEBUG ] 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:15:19 DEBUG "0" 08:15:19 DEBUG "pod": "ds-idrepo-1" 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG } 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG ] 08:15:19 DEBUG }, 08:15:19 DEBUG { 08:15:19 DEBUG ] 08:15:19 DEBUG { 08:15:19 DEBUG } 08:15:19 DEBUG "value": [ 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG } 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG ] 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "data": { 08:15:19 DEBUG } 08:15:19 DEBUG "data": { 08:15:19 DEBUG } 08:15:19 DEBUG "0" 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG "result": [ 08:15:19 DEBUG "result": [ 08:15:19 DEBUG } 08:15:19 DEBUG { 08:15:19 DEBUG { 08:15:19 DEBUG ] 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG } 08:15:19 DEBUG "pod": "ds-idrepo-2" 08:15:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:15:19 DEBUG } 08:15:19 DEBUG }, 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "ds-idrepo-1" 08:15:19 DEBUG "pod": "ds-cts-0" 08:15:19 DEBUG }, 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0" 08:15:19 DEBUG "0" 08:15:19 DEBUG ] 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "ds-cts-2" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "ds-idrepo-1" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "instance": "10.142.0.100:9100" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "instance": "10.142.0.99:9100" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "10932.833333333334" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0.1" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "ds-idrepo-2" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "ds-idrepo-1" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "instance": "10.142.0.100:9100" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "instance": "10.142.0.99:9100" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0.0003333333333333262" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "ds-cts-0" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0.14368258602989215" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "1.4219348249147958" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0.2" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "ds-cts-0" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0.07725555088297326" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "instance": "10.142.0.100:9100" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "instance": "10.142.0.99:9100" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0.00046666666666700016" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "14304858112" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "ds-cts-0" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "3413.4384597949415" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "pod": "ds-cts-2" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "pod": "ds-idrepo-1" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "0" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554119" 08:15:19 INFO [http_cmd]: http status code OK 08:15:19 DEBUG --- status code --- 08:15:19 DEBUG http status code is 200 (expected 200) 08:15:19 DEBUG --- http response --- 08:15:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:19 DEBUG ----- output ----- 08:15:19 DEBUG { 08:15:19 DEBUG "status": "success", 08:15:19 DEBUG "data": { 08:15:19 DEBUG "resultType": "vector", 08:15:19 DEBUG "result": [ 08:15:19 DEBUG { 08:15:19 DEBUG "metric": { 08:15:19 DEBUG "instance": "10.142.0.100:9100" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG [...] 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG 08:15:19 DEBUG "instance": "10.142.0.99:9100" 08:15:19 DEBUG }, 08:15:19 DEBUG "value": [ 08:15:19 DEBUG 1686554119, 08:15:19 DEBUG "5833.333333333333" 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG ] 08:15:19 DEBUG } 08:15:19 DEBUG } 08:15:53 INFO 08:15:53 INFO [loop_until]: kubectl --namespace=xlou top pods 08:15:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:15:53 INFO [loop_until]: OK (rc = 0) 08:15:53 DEBUG --- stdout --- 08:15:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 8m 6125Mi am-7849cf7bdb-npvzp 7m 6233Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 9m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 250m 13584Mi ds-idrepo-1 88m 13543Mi ds-idrepo-2 16m 13568Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5461Mi idm-58dc667486-vrqsf 5m 5521Mi lodemon-75f677b97d-6b58v 5m 71Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 982m 1103Mi 08:15:53 DEBUG --- stderr --- 08:15:53 DEBUG 08:16:05 INFO 08:16:05 INFO [loop_until]: kubectl --namespace=xlou top node 08:16:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:16:05 INFO [loop_until]: OK (rc = 0) 08:16:05 DEBUG --- stdout --- 08:16:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 61m 0% 6423Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 7161Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 66m 0% 7057Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 72m 0% 6413Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 67m 0% 7381Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 230m 1% 3054Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1206Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 67m 0% 1125Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1153Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 79m 0% 14534Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 156m 0% 14524Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1049Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 134m 0% 14576Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1044m 6% 2012Mi 3% 08:16:05 DEBUG --- stderr --- 08:16:05 DEBUG 127.0.0.1 - - [12/Jun/2023 08:16:12] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:16:18] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554179" 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "ds-idrepo-2" 08:16:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:16:19 DEBUG }, 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "ds-idrepo-1" 08:16:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:16:19 DEBUG }, 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "5426.866666666667" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG "0" 08:16:19 DEBUG ] 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG } 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "ds-idrepo-2" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "ds-idrepo-1" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "40158.1" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "ds-cts-2" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "ds-idrepo-1" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG }, 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG "value": [ 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG "0" 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:19 DEBUG ] 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG } 08:16:19 DEBUG { 08:16:19 DEBUG ] 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG } 08:16:19 DEBUG "data": { 08:16:19 DEBUG } 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "instance": "10.142.0.100:9100" 08:16:19 DEBUG "pod": "ds-cts-2" 08:16:19 DEBUG }, 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "instance": "10.142.0.99:9100" 08:16:19 DEBUG "pod": "ds-idrepo-1" 08:16:19 DEBUG }, 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "10794.3" 08:16:19 DEBUG "0" 08:16:19 DEBUG ] 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "ds-idrepo-2" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "ds-idrepo-1" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "33033.96666666667" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "instance": "10.142.0.100:9100" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "instance": "10.142.0.99:9100" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0.19225087379693062" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554179" 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0.8495896196620449" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0.2" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "ds-cts-1" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "instance": "10.142.0.100:9100" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "instance": "10.142.0.99:9100" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0.0003333333333330302" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG [...] 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:19 DEBUG 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:16:19 DEBUG { 08:16:19 DEBUG }, 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "value": [ 08:16:19 DEBUG "data": { 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "0" 08:16:19 DEBUG "result": [ 08:16:19 DEBUG ] 08:16:19 DEBUG { 08:16:19 DEBUG } 08:16:19 DEBUG "metric": { 08:16:19 DEBUG ] 08:16:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:16:19 DEBUG } 08:16:19 DEBUG }, 08:16:19 DEBUG } 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "14304759808" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "ds-cts-1" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "pod": "ds-cts-2" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "pod": "ds-idrepo-1" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "0" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 08:16:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554179" 08:16:19 INFO [http_cmd]: http status code OK 08:16:19 DEBUG --- status code --- 08:16:19 DEBUG http status code is 200 (expected 200) 08:16:19 DEBUG --- http response --- 08:16:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:19 DEBUG ----- output ----- 08:16:19 DEBUG { 08:16:19 DEBUG "status": "success", 08:16:19 DEBUG "data": { 08:16:19 DEBUG "resultType": "vector", 08:16:19 DEBUG "result": [ 08:16:19 DEBUG { 08:16:19 DEBUG "metric": { 08:16:19 DEBUG "instance": "10.142.0.100:9100" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG [...] 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG 08:16:19 DEBUG "instance": "10.142.0.99:9100" 08:16:19 DEBUG }, 08:16:19 DEBUG "value": [ 08:16:19 DEBUG 1686554179, 08:16:19 DEBUG "4619.633333333333" 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG ] 08:16:19 DEBUG } 08:16:19 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:16:25] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:16:31] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:16:38] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:16:44] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:16:50] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:16:53 INFO 08:16:53 INFO [loop_until]: kubectl --namespace=xlou top pods 08:16:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:16:53 INFO [loop_until]: OK (rc = 0) 08:16:53 DEBUG --- stdout --- 08:16:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 9m 6125Mi am-7849cf7bdb-npvzp 6m 6233Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 9m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 20m 13584Mi ds-idrepo-1 23m 13544Mi ds-idrepo-2 27m 13567Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5460Mi idm-58dc667486-vrqsf 5m 5521Mi lodemon-75f677b97d-6b58v 33m 76Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1201m 1184Mi 08:16:53 DEBUG --- stderr --- 08:16:53 DEBUG 127.0.0.1 - - [12/Jun/2023 08:16:57] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:17:03] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:17:05 INFO 08:17:05 INFO [loop_until]: kubectl --namespace=xlou top node 08:17:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:17:05 INFO [loop_until]: OK (rc = 0) 08:17:05 DEBUG --- stdout --- 08:17:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 65m 0% 6420Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 65m 0% 7161Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 7059Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 76m 0% 6414Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 66m 0% 7378Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 224m 1% 3047Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 114m 0% 1215Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1125Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1153Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 78m 0% 14536Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 90m 0% 14537Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 69m 0% 1060Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 74m 0% 14583Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1284m 8% 2090Mi 3% 08:17:05 DEBUG --- stderr --- 08:17:05 DEBUG 127.0.0.1 - - [12/Jun/2023 08:17:09] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:17:16] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "ds-idrepo-2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "ds-idrepo-1" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:19 DEBUG { 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG { 08:17:19 DEBUG { 08:17:19 DEBUG "data": { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "data": { 08:17:19 DEBUG "result": [ 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG { 08:17:19 DEBUG "result": [ 08:17:19 DEBUG "result": [ 08:17:19 DEBUG "metric": { 08:17:19 DEBUG { 08:17:19 DEBUG { 08:17:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG }, 08:17:19 DEBUG "pod": "ds-cts-2" 08:17:19 DEBUG "pod": "ds-idrepo-2" 08:17:19 DEBUG "value": [ 08:17:19 DEBUG }, 08:17:19 DEBUG }, 08:17:19 DEBUG 08:17:19 DEBUG "value": [ 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG }, 08:17:19 DEBUG "pod": "ds-idrepo-1" 08:17:19 DEBUG "pod": "ds-idrepo-1" 08:17:19 DEBUG "value": [ 08:17:19 DEBUG }, 08:17:19 DEBUG }, 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG "value": [ 08:17:19 DEBUG "0" 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG ] 08:17:19 DEBUG "0" 08:17:19 DEBUG "0" 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG ] 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554239" 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:19 DEBUG { 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG { 08:17:19 DEBUG "data": { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "result": [ 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG { 08:17:19 DEBUG "result": [ 08:17:19 DEBUG "metric": { 08:17:19 DEBUG { 08:17:19 DEBUG "pod": "ds-cts-2" 08:17:19 DEBUG "metric": { 08:17:19 DEBUG }, 08:17:19 DEBUG "instance": "10.142.0.100:9100" 08:17:19 DEBUG "value": [ 08:17:19 DEBUG }, 08:17:19 DEBUG 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "ds-idrepo-1" 08:17:19 DEBUG 08:17:19 DEBUG }, 08:17:19 DEBUG "instance": "10.142.0.99:9100" 08:17:19 DEBUG "value": [ 08:17:19 DEBUG }, 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG "0" 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG ] 08:17:19 DEBUG "10503.900000000001" 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "ds-idrepo-2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "ds-idrepo-1" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "1.032804219963353" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "instance": "10.142.0.100:9100" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "instance": "10.142.0.99:9100" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0.2" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0.00856588933737981" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "instance": "10.142.0.100:9100" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "instance": "10.142.0.99:9100" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0.00026666666666604516" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "0" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "14305714176" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554239" 08:17:19 INFO [http_cmd]: http status code OK 08:17:19 DEBUG --- status code --- 08:17:19 DEBUG http status code is 200 (expected 200) 08:17:19 DEBUG --- http response --- 08:17:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:17:19 DEBUG ----- output ----- 08:17:19 DEBUG { 08:17:19 DEBUG "status": "success", 08:17:19 DEBUG "data": { 08:17:19 DEBUG "resultType": "vector", 08:17:19 DEBUG "result": [ 08:17:19 DEBUG { 08:17:19 DEBUG "metric": { 08:17:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG [...] 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG 08:17:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:17:19 DEBUG }, 08:17:19 DEBUG "value": [ 08:17:19 DEBUG 1686554239, 08:17:19 DEBUG "4408.7453750420445" 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG ] 08:17:19 DEBUG } 08:17:19 DEBUG } 08:17:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554240" 08:17:20 INFO [http_cmd]: http status code OK 08:17:20 DEBUG --- status code --- 08:17:20 DEBUG http status code is 200 (expected 200) 08:17:20 DEBUG --- http response --- 08:17:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:17:20 DEBUG ----- output ----- 08:17:20 DEBUG { 08:17:20 DEBUG "status": "success", 08:17:20 DEBUG "data": { 08:17:20 DEBUG "resultType": "vector", 08:17:20 DEBUG "result": [ 08:17:20 DEBUG { 08:17:20 DEBUG "metric": { 08:17:20 DEBUG "pod": "ds-cts-2" 08:17:20 DEBUG }, 08:17:20 DEBUG "value": [ 08:17:20 DEBUG 08:17:20 DEBUG 08:17:20 DEBUG 08:17:20 DEBUG [...] 08:17:20 DEBUG 08:17:20 DEBUG 08:17:20 DEBUG 08:17:20 DEBUG "pod": "ds-idrepo-1" 08:17:20 DEBUG }, 08:17:20 DEBUG "value": [ 08:17:20 DEBUG 1686554240, 08:17:20 DEBUG "0" 08:17:20 DEBUG ] 08:17:20 DEBUG } 08:17:20 DEBUG ] 08:17:20 DEBUG } 08:17:20 DEBUG } 08:17:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554240" 08:17:20 INFO [http_cmd]: http status code OK 08:17:20 DEBUG --- status code --- 08:17:20 DEBUG http status code is 200 (expected 200) 08:17:20 DEBUG --- http response --- 08:17:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:20 DEBUG ----- output ----- 08:17:20 DEBUG { 08:17:20 DEBUG "status": "success", 08:17:20 DEBUG "data": { 08:17:20 DEBUG "resultType": "vector", 08:17:20 DEBUG "result": [ 08:17:20 DEBUG { 08:17:20 DEBUG "metric": { 08:17:20 DEBUG "instance": "10.142.0.100:9100" 08:17:20 DEBUG }, 08:17:20 DEBUG "value": [ 08:17:20 DEBUG 08:17:20 DEBUG 08:17:20 DEBUG 08:17:20 DEBUG [...] 08:17:20 DEBUG 08:17:20 DEBUG 08:17:20 DEBUG 08:17:20 DEBUG "instance": "10.142.0.99:9100" 08:17:20 DEBUG }, 08:17:20 DEBUG "value": [ 08:17:20 DEBUG 1686554240, 08:17:20 DEBUG "4580.7" 08:17:20 DEBUG ] 08:17:20 DEBUG } 08:17:20 DEBUG ] 08:17:20 DEBUG } 08:17:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:17:22] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:17:29] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:17:35] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:17:41] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:17:48] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:17:53 INFO 08:17:53 INFO [loop_until]: kubectl --namespace=xlou top pods 08:17:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:17:53 INFO [loop_until]: OK (rc = 0) 08:17:53 DEBUG --- stdout --- 08:17:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 9m 6125Mi am-7849cf7bdb-npvzp 6m 6233Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 9m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 17m 13584Mi ds-idrepo-1 43m 13544Mi ds-idrepo-2 17m 13568Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5460Mi idm-58dc667486-vrqsf 5m 5521Mi lodemon-75f677b97d-6b58v 46m 79Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1010m 1420Mi 08:17:53 DEBUG --- stderr --- 08:17:53 DEBUG 127.0.0.1 - - [12/Jun/2023 08:17:54] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:18:01] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:18:06 INFO 08:18:06 INFO [loop_until]: kubectl --namespace=xlou top node 08:18:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:18:06 INFO [loop_until]: OK (rc = 0) 08:18:06 DEBUG --- stdout --- 08:18:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 64m 0% 6419Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 7164Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 7061Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 71m 0% 6414Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 67m 0% 7379Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 226m 1% 3042Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 125m 0% 1219Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 72m 0% 1123Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1154Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 85m 0% 14540Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 298m 1% 14520Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 69m 0% 1048Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14578Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1077m 6% 2323Mi 3% 08:18:06 DEBUG --- stderr --- 08:18:06 DEBUG 127.0.0.1 - - [12/Jun/2023 08:18:07] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:18:13] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "ds-idrepo-2" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "ds-idrepo-1" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "ds-idrepo-2" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "ds-idrepo-1" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "ds-cts-2" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "ds-idrepo-1" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554299" 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "instance": "10.142.0.100:9100" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "instance": "10.142.0.99:9100" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "10461.2" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "ds-cts-2" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "ds-idrepo-1" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "ds-idrepo-2" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "ds-idrepo-1" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554299" 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG { 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG "data": { 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "data": { 08:18:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG }, 08:18:19 DEBUG "result": [ 08:18:19 DEBUG "value": [ 08:18:19 DEBUG { 08:18:19 DEBUG 08:18:19 DEBUG "metric": { 08:18:19 DEBUG 08:18:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:18:19 DEBUG 08:18:19 DEBUG }, 08:18:19 DEBUG [...] 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:18:19 DEBUG [...] 08:18:19 DEBUG }, 08:18:19 DEBUG 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG 08:18:19 DEBUG "0" 08:18:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:18:19 DEBUG ] 08:18:19 DEBUG }, 08:18:19 DEBUG } 08:18:19 DEBUG "value": [ 08:18:19 DEBUG ] 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG } 08:18:19 DEBUG "1.0305518224981087" 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "instance": "10.142.0.100:9100" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "instance": "10.142.0.99:9100" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0.2" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0.00368719049813985" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "instance": "10.142.0.100:9100" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "instance": "10.142.0.99:9100" 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0.0002999999999985903" 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554299" 08:18:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554299" 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 INFO [http_cmd]: http status code OK 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG --- status code --- 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG http status code is 200 (expected 200) 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG --- http response --- 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG ----- output ----- 08:18:19 DEBUG { 08:18:19 DEBUG { 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "status": "success", 08:18:19 DEBUG "data": { 08:18:19 DEBUG "data": { 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "resultType": "vector", 08:18:19 DEBUG "result": [ 08:18:19 DEBUG "result": [ 08:18:19 DEBUG { 08:18:19 DEBUG { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "metric": { 08:18:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:18:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:18:19 DEBUG }, 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG [...] 08:18:19 DEBUG [...] 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG 08:18:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:18:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:18:19 DEBUG }, 08:18:19 DEBUG }, 08:18:19 DEBUG "value": [ 08:18:19 DEBUG "value": [ 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG 1686554299, 08:18:19 DEBUG "0" 08:18:19 DEBUG "14307057664" 08:18:19 DEBUG ] 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 DEBUG ] 08:18:19 DEBUG ] 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:19 DEBUG } 08:18:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554300" 08:18:20 INFO [http_cmd]: http status code OK 08:18:20 DEBUG --- status code --- 08:18:20 DEBUG http status code is 200 (expected 200) 08:18:20 DEBUG --- http response --- 08:18:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:18:20 DEBUG ----- output ----- 08:18:20 DEBUG { 08:18:20 DEBUG "status": "success", 08:18:20 DEBUG "data": { 08:18:20 DEBUG "resultType": "vector", 08:18:20 DEBUG "result": [ 08:18:20 DEBUG { 08:18:20 DEBUG "metric": { 08:18:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:18:20 DEBUG }, 08:18:20 DEBUG "value": [ 08:18:20 DEBUG 08:18:20 DEBUG 08:18:20 DEBUG 08:18:20 DEBUG [...] 08:18:20 DEBUG 08:18:20 DEBUG 08:18:20 DEBUG 08:18:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:18:20 DEBUG }, 08:18:20 DEBUG "value": [ 08:18:20 DEBUG 1686554300, 08:18:20 DEBUG "0" 08:18:20 DEBUG ] 08:18:20 DEBUG } 08:18:20 DEBUG ] 08:18:20 DEBUG } 08:18:20 DEBUG } 08:18:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554300" 08:18:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554300" 08:18:20 INFO [http_cmd]: http status code OK 08:18:20 DEBUG --- status code --- 08:18:20 DEBUG http status code is 200 (expected 200) 08:18:20 DEBUG --- http response --- 08:18:20 INFO [http_cmd]: http status code OK 08:18:20 DEBUG --- status code --- 08:18:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:18:20 DEBUG http status code is 200 (expected 200) 08:18:20 DEBUG ----- output ----- 08:18:20 DEBUG --- http response --- 08:18:20 DEBUG { 08:18:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:20 DEBUG "status": "success", 08:18:20 DEBUG ----- output ----- 127.0.0.1 - - [12/Jun/2023 08:18:20] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:18:20 DEBUG "data": { 08:18:20 DEBUG { 08:18:20 DEBUG "resultType": "vector", 08:18:20 DEBUG "status": "success", 08:18:20 DEBUG "result": [ 08:18:20 DEBUG "data": { 08:18:20 DEBUG { 08:18:20 DEBUG "resultType": "vector", 08:18:20 DEBUG "metric": { 08:18:20 DEBUG "result": [ 08:18:20 DEBUG "pod": "ds-cts-2" 08:18:20 DEBUG { 08:18:20 DEBUG }, 08:18:20 DEBUG "metric": { 08:18:20 DEBUG "value": [ 08:18:20 DEBUG "instance": "10.142.0.100:9100" 08:18:20 DEBUG 08:18:20 DEBUG }, 08:18:20 DEBUG 08:18:20 DEBUG "value": [ 08:18:20 DEBUG 08:18:20 DEBUG 08:18:20 DEBUG [...] 08:18:20 DEBUG 08:18:20 DEBUG 08:18:20 DEBUG 08:18:20 DEBUG 08:18:20 DEBUG [...] 08:18:20 DEBUG 08:18:20 DEBUG 08:18:20 DEBUG "pod": "ds-idrepo-1" 08:18:20 DEBUG 08:18:20 DEBUG }, 08:18:20 DEBUG 08:18:20 DEBUG "value": [ 08:18:20 DEBUG "instance": "10.142.0.99:9100" 08:18:20 DEBUG 1686554300, 08:18:20 DEBUG }, 08:18:20 DEBUG "0" 08:18:20 DEBUG "value": [ 08:18:20 DEBUG ] 08:18:20 DEBUG 1686554300, 08:18:20 DEBUG } 08:18:20 DEBUG "6392.566666666666" 08:18:20 DEBUG ] 08:18:20 DEBUG ] 08:18:20 DEBUG } 08:18:20 DEBUG } 08:18:20 DEBUG } 08:18:20 DEBUG ] 08:18:20 DEBUG } 08:18:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:18:26] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:18:33] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:18:39] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:18:45] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:18:52] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:18:53 INFO 08:18:53 INFO [loop_until]: kubectl --namespace=xlou top pods 08:18:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:18:53 INFO [loop_until]: OK (rc = 0) 08:18:53 DEBUG --- stdout --- 08:18:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 8m 6125Mi am-7849cf7bdb-npvzp 6m 6233Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 9m 416Mi ds-cts-1 9m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 26m 13584Mi ds-idrepo-1 18m 13544Mi ds-idrepo-2 17m 13567Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5460Mi idm-58dc667486-vrqsf 5m 5521Mi lodemon-75f677b97d-6b58v 41m 79Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1021m 1650Mi 08:18:53 DEBUG --- stderr --- 08:18:53 DEBUG 127.0.0.1 - - [12/Jun/2023 08:18:58] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:19:05] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:19:06 INFO 08:19:06 INFO [loop_until]: kubectl --namespace=xlou top node 08:19:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:19:06 INFO [loop_until]: OK (rc = 0) 08:19:06 DEBUG --- stdout --- 08:19:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 66m 0% 6420Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 65m 0% 7164Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 63m 0% 7056Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 71m 0% 6416Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 7391Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 230m 1% 3051Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 113m 0% 1221Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1119Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1150Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 77m 0% 14537Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 79m 0% 14520Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 68m 0% 1046Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 83m 0% 14582Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1075m 6% 2554Mi 4% 08:19:06 DEBUG --- stderr --- 08:19:06 DEBUG 127.0.0.1 - - [12/Jun/2023 08:19:11] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:19:17] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "ds-idrepo-2" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "ds-idrepo-1" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG { 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG "data": { 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "data": { 08:19:19 DEBUG "instance": "10.142.0.100:9100" 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG }, 08:19:19 DEBUG "result": [ 08:19:19 DEBUG "value": [ 08:19:19 DEBUG { 08:19:19 DEBUG 08:19:19 DEBUG "metric": { 08:19:19 DEBUG 08:19:19 DEBUG "pod": "ds-idrepo-2" 08:19:19 DEBUG 08:19:19 DEBUG }, 08:19:19 DEBUG [...] 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "instance": "10.142.0.99:9100" 08:19:19 DEBUG [...] 08:19:19 DEBUG }, 08:19:19 DEBUG 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG 08:19:19 DEBUG "10702.366666666667" 08:19:19 DEBUG "pod": "ds-idrepo-1" 08:19:19 DEBUG ] 08:19:19 DEBUG }, 08:19:19 DEBUG } 08:19:19 DEBUG "value": [ 08:19:19 DEBUG ] 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG } 08:19:19 DEBUG "0" 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "ds-cts-2" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "ds-idrepo-1" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "ds-cts-2" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "ds-idrepo-1" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "ds-idrepo-2" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "ds-idrepo-1" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "1.0336343949082596" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [...] 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:19 DEBUG 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG 08:19:19 DEBUG { 08:19:19 DEBUG 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:19:19 DEBUG "data": { 08:19:19 DEBUG }, 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "value": [ 08:19:19 DEBUG "result": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG { 08:19:19 DEBUG "0" 08:19:19 DEBUG "metric": { 08:19:19 DEBUG ] 08:19:19 DEBUG "instance": "10.142.0.100:9100" 08:19:19 DEBUG } 08:19:19 DEBUG }, 08:19:19 DEBUG ] 08:19:19 DEBUG "value": [ 08:19:19 DEBUG } 08:19:19 DEBUG 08:19:19 DEBUG } 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "instance": "10.142.0.99:9100" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0.2" 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554359" 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG { 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG "data": { 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "data": { 08:19:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG }, 08:19:19 DEBUG "result": [ 08:19:19 DEBUG "value": [ 08:19:19 DEBUG { 08:19:19 DEBUG 08:19:19 DEBUG "metric": { 08:19:19 DEBUG 08:19:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:19:19 DEBUG 08:19:19 DEBUG }, 08:19:19 DEBUG [...] 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:19:19 DEBUG [...] 08:19:19 DEBUG }, 08:19:19 DEBUG 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG 08:19:19 DEBUG "1.3428799635496311" 08:19:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:19:19 DEBUG ] 08:19:19 DEBUG }, 08:19:19 DEBUG } 08:19:19 DEBUG "value": [ 08:19:19 DEBUG ] 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG } 08:19:19 DEBUG "14305722368" 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554359" 08:19:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554359" 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 INFO [http_cmd]: http status code OK 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG --- status code --- 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG http status code is 200 (expected 200) 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG --- http response --- 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG ----- output ----- 08:19:19 DEBUG { 08:19:19 DEBUG { 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "status": "success", 08:19:19 DEBUG "data": { 08:19:19 DEBUG "data": { 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "resultType": "vector", 08:19:19 DEBUG "result": [ 08:19:19 DEBUG "result": [ 08:19:19 DEBUG { 08:19:19 DEBUG { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "metric": { 08:19:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:19:19 DEBUG "instance": "10.142.0.100:9100" 08:19:19 DEBUG }, 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG [...] 08:19:19 DEBUG [...] 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG 08:19:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:19:19 DEBUG "instance": "10.142.0.99:9100" 08:19:19 DEBUG }, 08:19:19 DEBUG }, 08:19:19 DEBUG "value": [ 08:19:19 DEBUG "value": [ 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG 1686554359, 08:19:19 DEBUG "0" 08:19:19 DEBUG "0.0004333333333325603" 08:19:19 DEBUG ] 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 DEBUG ] 08:19:19 DEBUG ] 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:19 DEBUG } 08:19:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554360" 08:19:20 INFO [http_cmd]: http status code OK 08:19:20 DEBUG --- status code --- 08:19:20 DEBUG http status code is 200 (expected 200) 08:19:20 DEBUG --- http response --- 08:19:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:19:20 DEBUG ----- output ----- 08:19:20 DEBUG { 08:19:20 DEBUG "status": "success", 08:19:20 DEBUG "data": { 08:19:20 DEBUG "resultType": "vector", 08:19:20 DEBUG "result": [ 08:19:20 DEBUG { 08:19:20 DEBUG "metric": { 08:19:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:19:20 DEBUG }, 08:19:20 DEBUG "value": [ 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG [...] 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:19:20 DEBUG }, 08:19:20 DEBUG "value": [ 08:19:20 DEBUG 1686554360, 08:19:20 DEBUG "294407.82174080855" 08:19:20 DEBUG ] 08:19:20 DEBUG } 08:19:20 DEBUG ] 08:19:20 DEBUG } 08:19:20 DEBUG } 08:19:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554360" 08:19:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554360" 08:19:20 INFO [http_cmd]: http status code OK 08:19:20 INFO [http_cmd]: http status code OK 08:19:20 DEBUG --- status code --- 08:19:20 DEBUG --- status code --- 08:19:20 DEBUG http status code is 200 (expected 200) 08:19:20 DEBUG http status code is 200 (expected 200) 08:19:20 DEBUG --- http response --- 08:19:20 DEBUG --- http response --- 08:19:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:19:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:20 DEBUG ----- output ----- 08:19:20 DEBUG ----- output ----- 08:19:20 DEBUG { 08:19:20 DEBUG { 08:19:20 DEBUG "status": "success", 08:19:20 DEBUG "status": "success", 08:19:20 DEBUG "data": { 08:19:20 DEBUG "data": { 08:19:20 DEBUG "resultType": "vector", 08:19:20 DEBUG "resultType": "vector", 08:19:20 DEBUG "result": [ 08:19:20 DEBUG "result": [ 08:19:20 DEBUG { 08:19:20 DEBUG { 08:19:20 DEBUG "metric": { 08:19:20 DEBUG "metric": { 08:19:20 DEBUG "pod": "ds-cts-2" 08:19:20 DEBUG "instance": "10.142.0.100:9100" 08:19:20 DEBUG }, 08:19:20 DEBUG }, 08:19:20 DEBUG "value": [ 08:19:20 DEBUG "value": [ 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG [...] 08:19:20 DEBUG [...] 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG 08:19:20 DEBUG "pod": "ds-idrepo-1" 08:19:20 DEBUG "instance": "10.142.0.99:9100" 08:19:20 DEBUG }, 08:19:20 DEBUG }, 08:19:20 DEBUG "value": [ 08:19:20 DEBUG "value": [ 08:19:20 DEBUG 1686554360, 08:19:20 DEBUG 1686554360, 08:19:20 DEBUG "0" 08:19:20 DEBUG "4504.766666666666" 08:19:20 DEBUG ] 08:19:20 DEBUG ] 08:19:20 DEBUG } 08:19:20 DEBUG } 08:19:20 DEBUG ] 08:19:20 DEBUG ] 08:19:20 DEBUG } 08:19:20 DEBUG } 08:19:20 DEBUG } 08:19:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:19:24] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:19:30] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:19:37] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:19:43] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:19:49] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:19:53 INFO 08:19:53 INFO [loop_until]: kubectl --namespace=xlou top pods 08:19:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:19:53 INFO [loop_until]: OK (rc = 0) 08:19:53 DEBUG --- stdout --- 08:19:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 9m 6125Mi am-7849cf7bdb-npvzp 6m 6233Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 9m 370Mi ds-idrepo-0 17m 13584Mi ds-idrepo-1 17m 13544Mi ds-idrepo-2 17m 13568Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 9m 5460Mi idm-58dc667486-vrqsf 5m 5520Mi lodemon-75f677b97d-6b58v 45m 79Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1001m 1890Mi 08:19:53 DEBUG --- stderr --- 08:19:53 DEBUG 127.0.0.1 - - [12/Jun/2023 08:19:56] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:03] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:20:06 INFO 08:20:06 INFO [loop_until]: kubectl --namespace=xlou top node 08:20:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:20:06 INFO [loop_until]: OK (rc = 0) 08:20:06 DEBUG --- stdout --- 08:20:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 64m 0% 6420Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 7165Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 7056Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 71m 0% 6414Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 63m 0% 7376Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 217m 1% 3051Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 126m 0% 1222Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 71m 0% 1121Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1151Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 76m 0% 14541Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 77m 0% 14522Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 68m 0% 1051Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14583Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1075m 6% 2778Mi 4% 08:20:06 DEBUG --- stderr --- 08:20:06 DEBUG 127.0.0.1 - - [12/Jun/2023 08:20:09] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:15] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:16] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "ds-idrepo-2" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "pod": "ds-idrepo-1" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554419" 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG { 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "data": { 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG "result": [ 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG { 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "instance": "10.142.0.100:9100" 08:20:19 DEBUG "pod": "ds-idrepo-2" 08:20:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:20:19 DEBUG }, 08:20:19 DEBUG }, 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG "value": [ 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG [...] 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "instance": "10.142.0.99:9100" 08:20:19 DEBUG "pod": "ds-idrepo-1" 08:20:19 DEBUG "pod": "ds-cts-1" 08:20:19 DEBUG }, 08:20:19 DEBUG }, 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG "value": [ 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "10405.2" 08:20:19 DEBUG "0" 08:20:19 DEBUG "0" 08:20:19 DEBUG ] 08:20:19 DEBUG ] 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG ] 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "ds-cts-2" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "pod": "ds-idrepo-1" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "ds-cts-2" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "pod": "ds-idrepo-1" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "ds-idrepo-2" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "pod": "ds-idrepo-1" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0.6982063065507866" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG [...] 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:19 DEBUG 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG "pod": "ds-cts-1" 08:20:19 DEBUG { 08:20:19 DEBUG }, 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "value": [ 08:20:19 DEBUG "data": { 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "0" 08:20:19 DEBUG "result": [ 08:20:19 DEBUG ] 08:20:19 DEBUG { 08:20:19 DEBUG } 08:20:19 DEBUG "metric": { 08:20:19 DEBUG ] 08:20:19 DEBUG "instance": "10.142.0.100:9100" 08:20:19 DEBUG } 08:20:19 DEBUG }, 08:20:19 DEBUG } 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "instance": "10.142.0.99:9100" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0.2" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554419" 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554419" 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554419" 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG { 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "data": { 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG "result": [ 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG { 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:20:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:20:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:20:19 DEBUG }, 08:20:19 DEBUG }, 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG "value": [ 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG [...] 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "pod": "ds-cts-0" 08:20:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:20:19 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:20:19 DEBUG }, 08:20:19 DEBUG }, 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG "value": [ 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0" 08:20:19 DEBUG "0" 08:20:19 DEBUG "14307909632" 08:20:19 DEBUG ] 08:20:19 DEBUG ] 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG ] 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:19 INFO [http_cmd]: http status code OK 08:20:19 DEBUG --- status code --- 08:20:19 DEBUG http status code is 200 (expected 200) 08:20:19 DEBUG --- http response --- 08:20:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:19 DEBUG ----- output ----- 08:20:19 DEBUG { 08:20:19 DEBUG "status": "success", 08:20:19 DEBUG "data": { 08:20:19 DEBUG "resultType": "vector", 08:20:19 DEBUG "result": [ 08:20:19 DEBUG { 08:20:19 DEBUG "metric": { 08:20:19 DEBUG "instance": "10.142.0.100:9100" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG [...] 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG 08:20:19 DEBUG "instance": "10.142.0.99:9100" 08:20:19 DEBUG }, 08:20:19 DEBUG "value": [ 08:20:19 DEBUG 1686554419, 08:20:19 DEBUG "0.00036666666666747006" 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG ] 08:20:19 DEBUG } 08:20:19 DEBUG } 08:20:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554420" 08:20:20 INFO [http_cmd]: http status code OK 08:20:20 DEBUG --- status code --- 08:20:20 DEBUG http status code is 200 (expected 200) 08:20:20 DEBUG --- http response --- 08:20:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:20:20 DEBUG ----- output ----- 08:20:20 DEBUG { 08:20:20 DEBUG "status": "success", 08:20:20 DEBUG "data": { 08:20:20 DEBUG "resultType": "vector", 08:20:20 DEBUG "result": [ 08:20:20 DEBUG { 08:20:20 DEBUG "metric": { 08:20:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:20:20 DEBUG }, 08:20:20 DEBUG "value": [ 08:20:20 DEBUG 08:20:20 DEBUG 08:20:20 DEBUG 08:20:20 DEBUG [...] 08:20:20 DEBUG 08:20:20 DEBUG 08:20:20 DEBUG 08:20:20 DEBUG "pod": "ds-cts-0" 08:20:20 DEBUG }, 08:20:20 DEBUG "value": [ 08:20:20 DEBUG 1686554420, 08:20:20 DEBUG "0" 08:20:20 DEBUG ] 08:20:20 DEBUG } 08:20:20 DEBUG ] 08:20:20 DEBUG } 08:20:20 DEBUG } 08:20:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554420" 08:20:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554420" 08:20:20 INFO [http_cmd]: http status code OK 08:20:20 DEBUG --- status code --- 08:20:20 DEBUG http status code is 200 (expected 200) 08:20:20 DEBUG --- http response --- 08:20:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:20:20 DEBUG ----- output ----- 08:20:20 DEBUG { 08:20:20 INFO [http_cmd]: http status code OK 08:20:20 DEBUG "status": "success", 08:20:20 DEBUG --- status code --- 08:20:20 DEBUG "data": { 08:20:20 DEBUG http status code is 200 (expected 200) 08:20:20 DEBUG "resultType": "vector", 08:20:20 DEBUG --- http response --- 08:20:20 DEBUG "result": [ 08:20:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:20 DEBUG { 08:20:20 DEBUG ----- output ----- 08:20:20 DEBUG "metric": { 08:20:20 DEBUG { 08:20:20 DEBUG "pod": "ds-cts-2" 08:20:20 DEBUG "status": "success", 08:20:20 DEBUG }, 08:20:20 DEBUG "data": { 08:20:20 DEBUG "value": [ 08:20:20 DEBUG "resultType": "vector", 08:20:20 DEBUG 08:20:20 DEBUG "result": [ 08:20:20 DEBUG 08:20:20 DEBUG { 08:20:20 DEBUG 08:20:20 DEBUG "metric": { 08:20:20 DEBUG [...] 08:20:20 DEBUG "instance": "10.142.0.100:9100" 08:20:20 DEBUG 08:20:20 DEBUG }, 08:20:20 DEBUG 08:20:20 DEBUG "value": [ 08:20:20 DEBUG 08:20:20 DEBUG 08:20:20 DEBUG "pod": "ds-idrepo-1" 08:20:20 DEBUG 08:20:20 DEBUG }, 08:20:20 DEBUG 08:20:20 DEBUG "value": [ 08:20:20 DEBUG [...] 08:20:20 DEBUG 1686554420, 08:20:20 DEBUG 08:20:20 DEBUG "0" 08:20:20 DEBUG 08:20:20 DEBUG ] 08:20:20 DEBUG 08:20:20 DEBUG } 08:20:20 DEBUG "instance": "10.142.0.99:9100" 08:20:20 DEBUG ] 08:20:20 DEBUG }, 08:20:20 DEBUG } 08:20:20 DEBUG "value": [ 08:20:20 DEBUG } 08:20:20 DEBUG 1686554420, 08:20:20 DEBUG "6463.266666666666" 08:20:20 DEBUG ] 08:20:20 DEBUG } 08:20:20 DEBUG ] 08:20:20 DEBUG } 08:20:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:20:23] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:24] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:30] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:31] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:37] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:38] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:45] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:46] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:52] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:20:53] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:20:53 INFO 08:20:53 INFO [loop_until]: kubectl --namespace=xlou top pods 08:20:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:20:53 INFO [loop_until]: OK (rc = 0) 08:20:53 DEBUG --- stdout --- 08:20:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 9m 6125Mi am-7849cf7bdb-npvzp 7m 6233Mi am-7849cf7bdb-p4fns 9m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 9m 451Mi ds-cts-2 9m 370Mi ds-idrepo-0 16m 13584Mi ds-idrepo-1 18m 13544Mi ds-idrepo-2 19m 13567Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5459Mi idm-58dc667486-vrqsf 5m 5520Mi lodemon-75f677b97d-6b58v 66m 78Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 993m 1902Mi 08:20:53 DEBUG --- stderr --- 08:20:53 DEBUG 127.0.0.1 - - [12/Jun/2023 08:20:59] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:00] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:21:06 INFO 08:21:06 INFO [loop_until]: kubectl --namespace=xlou top node 08:21:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:21:06 INFO [loop_until]: OK (rc = 0) 08:21:06 DEBUG --- stdout --- 08:21:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 63m 0% 6420Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 65m 0% 7165Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 7053Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 72m 0% 6413Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 68m 0% 7376Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 236m 1% 3041Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 141m 0% 1223Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1122Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1152Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 80m 0% 14537Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 79m 0% 14524Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 67m 0% 1049Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14581Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1249m 7% 2851Mi 4% 08:21:06 DEBUG --- stderr --- 08:21:06 DEBUG 127.0.0.1 - - [12/Jun/2023 08:21:07] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:08] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:14] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:15] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554479" 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "pod": "ds-idrepo-2" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "pod": "ds-idrepo-1" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "instance": "10.142.0.100:9100" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "instance": "10.142.0.99:9100" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "10259.333333333334" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "pod": "ds-cts-2" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "pod": "ds-idrepo-1" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "pod": "ds-idrepo-2" 08:21:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:21:19 DEBUG }, 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "pod": "ds-idrepo-1" 08:21:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:21:19 DEBUG }, 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0" 08:21:19 DEBUG "0" 08:21:19 DEBUG ] 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "pod": "ds-cts-2" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "pod": "ds-idrepo-1" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "pod": "ds-idrepo-2" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "pod": "ds-idrepo-1" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554479" 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554479" 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG { 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG "data": { 08:21:19 DEBUG "data": { 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG "result": [ 08:21:19 DEBUG "result": [ 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:19 DEBUG { 08:21:19 DEBUG { 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG { 08:21:19 DEBUG "instance": "10.142.0.100:9100" 08:21:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG }, 08:21:19 DEBUG }, 08:21:19 DEBUG "data": { 08:21:19 DEBUG "value": [ 08:21:19 DEBUG "value": [ 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "result": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG { 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "metric": { 08:21:19 DEBUG [...] 08:21:19 DEBUG [...] 08:21:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG }, 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "instance": "10.142.0.99:9100" 08:21:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:21:19 DEBUG 08:21:19 DEBUG }, 08:21:19 DEBUG }, 08:21:19 DEBUG 08:21:19 DEBUG "value": [ 08:21:19 DEBUG "value": [ 08:21:19 DEBUG [...] 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG 08:21:19 DEBUG "0" 08:21:19 DEBUG "1.0443708380069312" 08:21:19 DEBUG 08:21:19 DEBUG ] 08:21:19 DEBUG ] 08:21:19 DEBUG 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:21:19 DEBUG ] 08:21:19 DEBUG ] 08:21:19 DEBUG }, 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 DEBUG "value": [ 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554479" 08:21:19 INFO [http_cmd]: http status code OK 08:21:19 DEBUG --- status code --- 08:21:19 DEBUG http status code is 200 (expected 200) 08:21:19 DEBUG --- http response --- 08:21:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:19 DEBUG ----- output ----- 08:21:19 DEBUG { 08:21:19 DEBUG "status": "success", 08:21:19 DEBUG "data": { 08:21:19 DEBUG "resultType": "vector", 08:21:19 DEBUG "result": [ 08:21:19 DEBUG { 08:21:19 DEBUG "metric": { 08:21:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG [...] 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG 08:21:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:21:19 DEBUG }, 08:21:19 DEBUG "value": [ 08:21:19 DEBUG 1686554479, 08:21:19 DEBUG "0.2" 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG ] 08:21:19 DEBUG } 08:21:19 DEBUG } 08:21:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554480" 08:21:20 INFO [http_cmd]: http status code OK 08:21:20 DEBUG --- status code --- 08:21:20 DEBUG http status code is 200 (expected 200) 08:21:20 DEBUG --- http response --- 08:21:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:20 DEBUG ----- output ----- 08:21:20 DEBUG { 08:21:20 DEBUG "status": "success", 08:21:20 DEBUG "data": { 08:21:20 DEBUG "resultType": "vector", 08:21:20 DEBUG "result": [ 08:21:20 DEBUG { 08:21:20 DEBUG "metric": { 08:21:20 DEBUG "instance": "10.142.0.100:9100" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG [...] 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG "instance": "10.142.0.99:9100" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 1686554480, 08:21:20 DEBUG "0.00030000000000048505" 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG } 08:21:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554480" 08:21:20 INFO [http_cmd]: http status code OK 08:21:20 DEBUG --- status code --- 08:21:20 DEBUG http status code is 200 (expected 200) 08:21:20 DEBUG --- http response --- 08:21:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:20 DEBUG ----- output ----- 08:21:20 DEBUG { 08:21:20 DEBUG "status": "success", 08:21:20 DEBUG "data": { 08:21:20 DEBUG "resultType": "vector", 08:21:20 DEBUG "result": [ 08:21:20 DEBUG { 08:21:20 DEBUG "metric": { 08:21:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG [...] 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 1686554480, 08:21:20 DEBUG "2.0249634491134474" 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG } 08:21:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554480" 08:21:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554480" 08:21:20 INFO [http_cmd]: http status code OK 08:21:20 DEBUG --- status code --- 08:21:20 DEBUG http status code is 200 (expected 200) 08:21:20 DEBUG --- http response --- 08:21:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:20 DEBUG ----- output ----- 08:21:20 DEBUG { 08:21:20 DEBUG "status": "success", 08:21:20 DEBUG "data": { 08:21:20 DEBUG "resultType": "vector", 08:21:20 DEBUG "result": [ 08:21:20 DEBUG { 08:21:20 DEBUG "metric": { 08:21:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG [...] 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 1686554480, 08:21:20 DEBUG "0" 08:21:20 INFO [http_cmd]: http status code OK 08:21:20 DEBUG ] 08:21:20 DEBUG --- status code --- 08:21:20 DEBUG } 08:21:20 DEBUG http status code is 200 (expected 200) 08:21:20 DEBUG ] 08:21:20 DEBUG --- http response --- 08:21:20 DEBUG } 08:21:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:20 DEBUG } 08:21:20 DEBUG ----- output ----- 08:21:20 DEBUG { 08:21:20 DEBUG "status": "success", 08:21:20 DEBUG "data": { 08:21:20 DEBUG "resultType": "vector", 08:21:20 DEBUG "result": [ 08:21:20 DEBUG { 08:21:20 DEBUG "metric": { 08:21:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG [...] 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 1686554480, 08:21:20 DEBUG "14308646912" 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG } 08:21:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554480" 08:21:20 INFO [http_cmd]: http status code OK 08:21:20 DEBUG --- status code --- 08:21:20 DEBUG http status code is 200 (expected 200) 08:21:20 DEBUG --- http response --- 08:21:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:20 DEBUG ----- output ----- 08:21:20 DEBUG { 08:21:20 DEBUG "status": "success", 08:21:20 DEBUG "data": { 08:21:20 DEBUG "resultType": "vector", 08:21:20 DEBUG "result": [ 08:21:20 DEBUG { 08:21:20 DEBUG "metric": { 08:21:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG [...] 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 1686554480, 08:21:20 DEBUG "387039310.43301415" 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG } 08:21:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554480" 08:21:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554480" 08:21:20 INFO [http_cmd]: http status code OK 08:21:20 DEBUG --- status code --- 08:21:20 DEBUG http status code is 200 (expected 200) 08:21:20 DEBUG --- http response --- 08:21:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:21:20 DEBUG ----- output ----- 08:21:20 DEBUG { 08:21:20 DEBUG "status": "success", 08:21:20 DEBUG "data": { 08:21:20 DEBUG "resultType": "vector", 08:21:20 DEBUG "result": [ 08:21:20 DEBUG { 08:21:20 DEBUG "metric": { 08:21:20 DEBUG "pod": "ds-cts-2" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG [...] 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG "pod": "ds-idrepo-1" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 1686554480, 08:21:20 DEBUG "0" 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG } 08:21:20 INFO [http_cmd]: http status code OK 08:21:20 DEBUG --- status code --- 08:21:20 DEBUG http status code is 200 (expected 200) 08:21:20 DEBUG --- http response --- 08:21:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:20 DEBUG ----- output ----- 08:21:20 DEBUG { 08:21:20 DEBUG "status": "success", 08:21:20 DEBUG "data": { 08:21:20 DEBUG "resultType": "vector", 08:21:20 DEBUG "result": [ 08:21:20 DEBUG { 08:21:20 DEBUG "metric": { 08:21:20 DEBUG "instance": "10.142.0.100:9100" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG [...] 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG 08:21:20 DEBUG "instance": "10.142.0.99:9100" 08:21:20 DEBUG }, 08:21:20 DEBUG "value": [ 08:21:20 DEBUG 1686554480, 08:21:20 DEBUG "4525.433333333333" 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG ] 08:21:20 DEBUG } 08:21:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:21:22] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:22] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:29] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:30] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:36] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:37] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:44] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:45] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:51] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:52] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:21:53 INFO 08:21:53 INFO [loop_until]: kubectl --namespace=xlou top pods 08:21:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:21:53 INFO [loop_until]: OK (rc = 0) 08:21:53 DEBUG --- stdout --- 08:21:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 9m 6125Mi am-7849cf7bdb-npvzp 6m 6233Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 9m 451Mi ds-cts-2 12m 370Mi ds-idrepo-0 18m 13585Mi ds-idrepo-1 18m 13544Mi ds-idrepo-2 16m 13568Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5459Mi idm-58dc667486-vrqsf 5m 5520Mi lodemon-75f677b97d-6b58v 65m 80Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1001m 2209Mi 08:21:53 DEBUG --- stderr --- 08:21:53 DEBUG 127.0.0.1 - - [12/Jun/2023 08:21:58] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:21:59] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:06] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:22:06 INFO 08:22:06 INFO [loop_until]: kubectl --namespace=xlou top node 08:22:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:22:06 INFO [loop_until]: OK (rc = 0) 08:22:06 DEBUG --- stdout --- 08:22:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 66m 0% 6417Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 7164Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 7057Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 71m 0% 6412Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 67m 0% 7378Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 231m 1% 3032Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 146m 0% 1219Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 73m 0% 1124Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1154Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 72m 0% 14535Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 74m 0% 14523Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 67m 0% 1045Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14583Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1078m 6% 3095Mi 5% 08:22:06 DEBUG --- stderr --- 08:22:06 DEBUG 127.0.0.1 - - [12/Jun/2023 08:22:06] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:13] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:14] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554539" 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "data": { 08:22:19 DEBUG "data": { 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "result": [ 08:22:19 DEBUG "result": [ 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "pod": "ds-idrepo-2" 08:22:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:22:19 DEBUG }, 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG [...] 08:22:19 DEBUG [...] 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG "pod": "ds-idrepo-1" 08:22:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 DEBUG }, 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG "0" 08:22:19 DEBUG "0" 08:22:19 DEBUG ] 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG ] 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 DEBUG } 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 DEBUG } 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG } 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "data": { 08:22:19 DEBUG "data": { 08:22:19 DEBUG "data": { 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "result": [ 08:22:19 DEBUG "result": [ 08:22:19 DEBUG "result": [ 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "pod": "ds-cts-2" 08:22:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:22:19 DEBUG "instance": "10.142.0.100:9100" 08:22:19 DEBUG }, 08:22:19 DEBUG }, 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG "value": [ 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG [...] 08:22:19 DEBUG [...] 08:22:19 DEBUG [...] 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG "pod": "ds-idrepo-1" 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:22:19 DEBUG "instance": "10.142.0.99:9100" 08:22:19 DEBUG }, 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG }, 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:19 DEBUG "value": [ 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG "0" 08:22:19 DEBUG { 08:22:19 DEBUG "0" 08:22:19 DEBUG "10576.4" 08:22:19 DEBUG ] 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG ] 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG "data": { 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG ] 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG ] 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG "result": [ 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG { 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "pod": "ds-idrepo-2" 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG [...] 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG "pod": "ds-idrepo-1" 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG "0" 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG { 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "data": { 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "result": [ 08:22:19 DEBUG { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG [...] 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG "0" 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "data": { 08:22:19 DEBUG "data": { 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "result": [ 08:22:19 DEBUG "result": [ 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "pod": "ds-idrepo-2" 08:22:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:22:19 DEBUG }, 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG [...] 08:22:19 DEBUG [...] 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG "pod": "ds-idrepo-1" 08:22:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:22:19 DEBUG }, 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG "0" 08:22:19 DEBUG "0" 08:22:19 DEBUG ] 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG ] 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG { 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "data": { 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "result": [ 08:22:19 DEBUG { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "pod": "ds-cts-2" 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG [...] 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG "pod": "ds-idrepo-1" 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG "0" 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554539" 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554539" 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "data": { 08:22:19 DEBUG "data": { 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "resultType": "vector", 08:22:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554539" 08:22:19 DEBUG "result": [ 08:22:19 DEBUG "result": [ 08:22:19 DEBUG { 08:22:19 DEBUG { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:22:19 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:22:19 DEBUG }, 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG [...] 08:22:19 DEBUG [...] 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:22:19 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:22:19 DEBUG }, 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG "0" 08:22:19 DEBUG "1.0249161788316528" 08:22:19 DEBUG ] 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG ] 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:19 INFO [http_cmd]: http status code OK 08:22:19 DEBUG --- status code --- 08:22:19 DEBUG http status code is 200 (expected 200) 08:22:19 DEBUG --- http response --- 08:22:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:19 DEBUG ----- output ----- 08:22:19 DEBUG { 08:22:19 DEBUG "status": "success", 08:22:19 DEBUG "data": { 08:22:19 DEBUG "resultType": "vector", 08:22:19 DEBUG "result": [ 08:22:19 DEBUG { 08:22:19 DEBUG "metric": { 08:22:19 DEBUG "instance": "10.142.0.100:9100" 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG [...] 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG 08:22:19 DEBUG "instance": "10.142.0.99:9100" 08:22:19 DEBUG }, 08:22:19 DEBUG "value": [ 08:22:19 DEBUG 1686554539, 08:22:19 DEBUG "0" 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG ] 08:22:19 DEBUG } 08:22:19 DEBUG } 08:22:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554540" 08:22:20 INFO [http_cmd]: http status code OK 08:22:20 DEBUG --- status code --- 08:22:20 DEBUG http status code is 200 (expected 200) 08:22:20 DEBUG --- http response --- 08:22:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:20 DEBUG ----- output ----- 08:22:20 DEBUG { 08:22:20 DEBUG "status": "success", 08:22:20 DEBUG "data": { 08:22:20 DEBUG "resultType": "vector", 08:22:20 DEBUG "result": [ 08:22:20 DEBUG { 08:22:20 DEBUG "metric": { 08:22:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG [...] 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 1686554540, 08:22:20 DEBUG "0.2" 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG } 08:22:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554540" 08:22:20 INFO [http_cmd]: http status code OK 08:22:20 DEBUG --- status code --- 08:22:20 DEBUG http status code is 200 (expected 200) 08:22:20 DEBUG --- http response --- 08:22:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:20 DEBUG ----- output ----- 08:22:20 DEBUG { 08:22:20 DEBUG "status": "success", 08:22:20 DEBUG "data": { 08:22:20 DEBUG "resultType": "vector", 08:22:20 DEBUG "result": [ 08:22:20 DEBUG { 08:22:20 DEBUG "metric": { 08:22:20 DEBUG "instance": "10.142.0.100:9100" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG [...] 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG "instance": "10.142.0.99:9100" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 1686554540, 08:22:20 DEBUG "0.00026666666666604516" 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG } 08:22:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554540" 08:22:20 INFO [http_cmd]: http status code OK 08:22:20 DEBUG --- status code --- 08:22:20 DEBUG http status code is 200 (expected 200) 08:22:20 DEBUG --- http response --- 08:22:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:22:20 DEBUG ----- output ----- 08:22:20 DEBUG { 08:22:20 DEBUG "status": "success", 08:22:20 DEBUG "data": { 08:22:20 DEBUG "resultType": "vector", 08:22:20 DEBUG "result": [ 08:22:20 DEBUG { 08:22:20 DEBUG "metric": { 08:22:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG [...] 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG "pod": "ds-cts-0" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 1686554540, 08:22:20 DEBUG "0" 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG } 08:22:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554540" 08:22:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554540" 08:22:20 INFO [http_cmd]: http status code OK 08:22:20 DEBUG --- status code --- 08:22:20 DEBUG http status code is 200 (expected 200) 08:22:20 DEBUG --- http response --- 08:22:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:20 DEBUG ----- output ----- 08:22:20 DEBUG { 08:22:20 DEBUG "status": "success", 08:22:20 DEBUG "data": { 08:22:20 DEBUG "resultType": "vector", 08:22:20 DEBUG "result": [ 08:22:20 DEBUG { 08:22:20 DEBUG "metric": { 08:22:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 08:22:20 INFO [http_cmd]: http status code OK 08:22:20 DEBUG 08:22:20 DEBUG --- status code --- 08:22:20 DEBUG 08:22:20 DEBUG http status code is 200 (expected 200) 08:22:20 DEBUG [...] 08:22:20 DEBUG --- http response --- 08:22:20 DEBUG 08:22:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:20 DEBUG 08:22:20 DEBUG ----- output ----- 08:22:20 DEBUG 08:22:20 DEBUG { 08:22:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:22:20 DEBUG "status": "success", 08:22:20 DEBUG }, 08:22:20 DEBUG "data": { 08:22:20 DEBUG "value": [ 08:22:20 DEBUG "resultType": "vector", 08:22:20 DEBUG 1686554540, 08:22:20 DEBUG "result": [ 08:22:20 DEBUG "0" 08:22:20 DEBUG { 08:22:20 DEBUG ] 08:22:20 DEBUG "metric": { 08:22:20 DEBUG } 08:22:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:22:20 DEBUG ] 08:22:20 DEBUG }, 08:22:20 DEBUG } 08:22:20 DEBUG "value": [ 08:22:20 DEBUG } 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG [...] 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 1686554540, 08:22:20 DEBUG "14306234368" 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG } 08:22:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554540" 08:22:20 INFO [http_cmd]: http status code OK 08:22:20 DEBUG --- status code --- 08:22:20 DEBUG http status code is 200 (expected 200) 08:22:20 DEBUG --- http response --- 08:22:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:22:20 DEBUG ----- output ----- 08:22:20 DEBUG { 08:22:20 DEBUG "status": "success", 08:22:20 DEBUG "data": { 08:22:20 DEBUG "resultType": "vector", 08:22:20 DEBUG "result": [ 08:22:20 DEBUG { 08:22:20 DEBUG "metric": { 08:22:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG [...] 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG "pod": "ds-cts-0" 08:22:20 DEBUG }, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 1686554540, 08:22:20 DEBUG "0" 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG } 08:22:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554540" 08:22:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554540" 08:22:20 INFO [http_cmd]: http status code OK 08:22:20 DEBUG --- status code --- 08:22:20 INFO [http_cmd]: http status code OK 127.0.0.1 - - [12/Jun/2023 08:22:20] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:22:20 DEBUG http status code is 200 (expected 200) 08:22:20 DEBUG --- status code --- 08:22:20 DEBUG --- http response --- 08:22:20 DEBUG http status code is 200 (expected 200) 08:22:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:22:20 DEBUG --- http response --- 08:22:20 DEBUG ----- output ----- 08:22:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:20 DEBUG { 08:22:20 DEBUG ----- output ----- 08:22:20 DEBUG "status": "success", 08:22:20 DEBUG { 08:22:20 DEBUG "data": { 08:22:20 DEBUG "status": "success", 08:22:20 DEBUG "resultType": "vector", 08:22:20 DEBUG "data": { 08:22:20 DEBUG "result": [ 08:22:20 DEBUG "resultType": "vector", 08:22:20 DEBUG { 08:22:20 DEBUG "result": [ 08:22:20 DEBUG "metric": { 08:22:20 DEBUG { 08:22:20 DEBUG "pod": "ds-cts-2" 08:22:20 DEBUG "metric": { 08:22:20 DEBUG }, 08:22:20 DEBUG "instance": "10.142.0.100:9100" 08:22:20 DEBUG "value": [ 08:22:20 DEBUG }, 08:22:20 DEBUG 08:22:20 DEBUG "value": [ 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG [...] 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG [...] 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG 08:22:20 DEBUG "pod": "ds-idrepo-1" 08:22:20 DEBUG 08:22:20 DEBUG }, 08:22:20 DEBUG "instance": "10.142.0.99:9100" 08:22:20 DEBUG "value": [ 08:22:20 DEBUG }, 08:22:20 DEBUG 1686554540, 08:22:20 DEBUG "value": [ 08:22:20 DEBUG "0" 08:22:20 DEBUG 1686554540, 08:22:20 DEBUG ] 08:22:20 DEBUG "3957.8999999999996" 08:22:20 DEBUG } 08:22:20 DEBUG ] 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG } 08:22:20 DEBUG ] 08:22:20 DEBUG } 08:22:20 DEBUG } 08:22:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:22:21] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:27] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:28] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:35] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:36] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:42] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:43] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:50] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:51] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:22:53 INFO 08:22:53 INFO [loop_until]: kubectl --namespace=xlou top pods 08:22:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:22:54 INFO [loop_until]: OK (rc = 0) 08:22:54 DEBUG --- stdout --- 08:22:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 9m 6125Mi am-7849cf7bdb-npvzp 8m 6234Mi am-7849cf7bdb-p4fns 7m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 9m 451Mi ds-cts-2 9m 370Mi ds-idrepo-0 16m 13585Mi ds-idrepo-1 18m 13544Mi ds-idrepo-2 18m 13567Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5459Mi idm-58dc667486-vrqsf 5m 5520Mi lodemon-75f677b97d-6b58v 58m 79Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1010m 2450Mi 08:22:54 DEBUG --- stderr --- 08:22:54 DEBUG 127.0.0.1 - - [12/Jun/2023 08:22:57] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:22:58] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:04] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:05] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:23:06 INFO 08:23:06 INFO [loop_until]: kubectl --namespace=xlou top node 08:23:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:23:06 INFO [loop_until]: OK (rc = 0) 08:23:06 DEBUG --- stdout --- 08:23:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 62m 0% 6415Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 7165Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 7054Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 71m 0% 6412Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 66m 0% 7379Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 231m 1% 3043Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 146m 0% 1217Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1122Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 66m 0% 1158Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 80m 0% 14535Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 83m 0% 14523Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 71m 0% 1049Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 74m 0% 14586Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1073m 6% 3336Mi 5% 08:23:06 DEBUG --- stderr --- 08:23:06 DEBUG 127.0.0.1 - - [12/Jun/2023 08:23:12] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:13] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:19] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554599" 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG { 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 DEBUG { 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG "data": { 08:23:19 DEBUG { 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 DEBUG "data": { 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG "result": [ 08:23:19 DEBUG "data": { 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:19 DEBUG { 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:23:19 DEBUG "result": [ 08:23:19 DEBUG { 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG { 08:23:19 DEBUG "metric": { 08:23:19 DEBUG "result": [ 08:23:19 DEBUG { 08:23:19 DEBUG "data": { 08:23:19 DEBUG { 08:23:19 DEBUG "metric": { 08:23:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:23:19 DEBUG { 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG "instance": "10.142.0.100:9100" 08:23:19 DEBUG }, 08:23:19 DEBUG "metric": { 08:23:19 DEBUG "data": { 08:23:19 DEBUG "result": [ 08:23:19 DEBUG "data": { 08:23:19 DEBUG }, 08:23:19 DEBUG "value": [ 08:23:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG { 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG "value": [ 08:23:19 DEBUG 08:23:19 DEBUG }, 08:23:19 DEBUG "result": [ 08:23:19 DEBUG "metric": { 08:23:19 DEBUG "result": [ 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG "value": [ 08:23:19 DEBUG { 08:23:19 DEBUG "pod": "ds-idrepo-2" 08:23:19 DEBUG { 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG "metric": { 08:23:19 DEBUG }, 08:23:19 DEBUG "metric": { 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 DEBUG 08:23:19 DEBUG [...] 08:23:19 DEBUG 08:23:19 DEBUG "pod": "ds-idrepo-2" 08:23:19 DEBUG "value": [ 08:23:19 DEBUG "pod": "ds-cts-2" 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG [...] 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG }, 08:23:19 DEBUG 08:23:19 DEBUG }, 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG [...] 08:23:19 DEBUG "value": [ 08:23:19 DEBUG 08:23:19 DEBUG "value": [ 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG { 08:23:19 DEBUG 08:23:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG [...] 08:23:19 DEBUG 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG "instance": "10.142.0.99:9100" 08:23:19 DEBUG }, 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG "data": { 08:23:19 DEBUG }, 08:23:19 DEBUG "value": [ 08:23:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:23:19 DEBUG [...] 08:23:19 DEBUG 08:23:19 DEBUG [...] 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG "value": [ 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG }, 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG "result": [ 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG "0" 08:23:19 DEBUG "value": [ 08:23:19 DEBUG 08:23:19 DEBUG "pod": "ds-idrepo-1" 08:23:19 DEBUG 08:23:19 DEBUG { 08:23:19 DEBUG "11316.966666666667" 08:23:19 DEBUG ] 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG 08:23:19 DEBUG }, 08:23:19 DEBUG 08:23:19 DEBUG "metric": { 08:23:19 DEBUG ] 08:23:19 DEBUG } 08:23:19 DEBUG "0" 08:23:19 DEBUG "pod": "ds-idrepo-1" 08:23:19 DEBUG "value": [ 08:23:19 DEBUG "pod": "ds-idrepo-1" 08:23:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:23:19 DEBUG } 08:23:19 DEBUG ] 08:23:19 DEBUG ] 08:23:19 DEBUG }, 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG }, 08:23:19 DEBUG }, 08:23:19 DEBUG ] 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG "value": [ 08:23:19 DEBUG "0" 08:23:19 DEBUG "value": [ 08:23:19 DEBUG "value": [ 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG ] 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG ] 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG "0" 08:23:19 DEBUG } 08:23:19 DEBUG "0" 08:23:19 DEBUG 08:23:19 DEBUG } 08:23:19 DEBUG ] 08:23:19 DEBUG ] 08:23:19 DEBUG ] 08:23:19 DEBUG 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG [...] 08:23:19 DEBUG ] 08:23:19 DEBUG } 08:23:19 DEBUG ] 08:23:19 DEBUG 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG 08:23:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:23:19 DEBUG }, 08:23:19 DEBUG "value": [ 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG "0" 08:23:19 DEBUG ] 08:23:19 DEBUG } 08:23:19 DEBUG ] 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG { 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG "data": { 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG "result": [ 08:23:19 DEBUG { 08:23:19 DEBUG "metric": { 08:23:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:23:19 DEBUG }, 08:23:19 DEBUG "value": [ 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG [...] 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:23:19 DEBUG }, 08:23:19 DEBUG "value": [ 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG "0" 08:23:19 DEBUG ] 08:23:19 DEBUG } 08:23:19 DEBUG ] 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 INFO [http_cmd]: http status code OK 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG --- status code --- 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 DEBUG http status code is 200 (expected 200) 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG --- http response --- 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG ----- output ----- 08:23:19 DEBUG { 08:23:19 DEBUG { 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG "status": "success", 08:23:19 DEBUG "data": { 08:23:19 DEBUG "data": { 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG "resultType": "vector", 08:23:19 DEBUG "result": [ 08:23:19 DEBUG "result": [ 08:23:19 DEBUG { 08:23:19 DEBUG { 08:23:19 DEBUG "metric": { 08:23:19 DEBUG "metric": { 08:23:19 DEBUG "pod": "ds-idrepo-2" 08:23:19 DEBUG "pod": "ds-cts-2" 08:23:19 DEBUG }, 08:23:19 DEBUG }, 08:23:19 DEBUG "value": [ 08:23:19 DEBUG "value": [ 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG [...] 08:23:19 DEBUG [...] 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG 08:23:19 DEBUG "pod": "ds-idrepo-1" 08:23:19 DEBUG "pod": "ds-idrepo-1" 08:23:19 DEBUG }, 08:23:19 DEBUG }, 08:23:19 DEBUG "value": [ 08:23:19 DEBUG "value": [ 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG 1686554599, 08:23:19 DEBUG "0" 08:23:19 DEBUG "0" 08:23:19 DEBUG ] 08:23:19 DEBUG ] 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG ] 08:23:19 DEBUG ] 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 DEBUG } 08:23:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554599" 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG { 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "data": { 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "result": [ 08:23:20 DEBUG { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG [...] 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 1686554599, 08:23:20 DEBUG "0" 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554600" 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554600" 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG { 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "data": { 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "result": [ 08:23:20 DEBUG { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG [...] 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG "1.0267805765231999" 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG { 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "data": { 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "result": [ 08:23:20 DEBUG { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "instance": "10.142.0.100:9100" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG [...] 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG "instance": "10.142.0.99:9100" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG "0" 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554600" 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554600" 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG { 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "data": { 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "result": [ 08:23:20 DEBUG { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG [...] 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG "0.2" 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG { 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "data": { 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "result": [ 08:23:20 DEBUG { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "instance": "10.142.0.100:9100" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG [...] 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG "instance": "10.142.0.99:9100" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG "0.00026666666666604516" 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554600" 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554600" 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG { 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "data": { 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "result": [ 08:23:20 DEBUG { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG [...] 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG "0" 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG { 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "data": { 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "result": [ 08:23:20 DEBUG { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG [...] 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG "0.006577000731997073" 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554600" 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG { 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "data": { 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "result": [ 08:23:20 DEBUG { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554600" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 INFO [http_cmd]: http status code OK 127.0.0.1 - - [12/Jun/2023 08:23:20] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:23:20 DEBUG 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [...] 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:23:20 DEBUG 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG 08:23:20 DEBUG { 08:23:20 DEBUG 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:23:20 DEBUG "data": { 08:23:20 DEBUG }, 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "value": [ 08:23:20 DEBUG "result": [ 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG { 08:23:20 DEBUG "14304669696" 08:23:20 DEBUG "metric": { 08:23:20 DEBUG ] 08:23:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:23:20 DEBUG } 08:23:20 DEBUG }, 08:23:20 DEBUG ] 08:23:20 DEBUG "value": [ 08:23:20 DEBUG } 08:23:20 DEBUG 08:23:20 DEBUG } 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG [...] 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG "3957.7196732351476" 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554600" 08:23:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554600" 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 INFO [http_cmd]: http status code OK 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG --- status code --- 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG http status code is 200 (expected 200) 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG --- http response --- 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:23:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG ----- output ----- 08:23:20 DEBUG { 08:23:20 DEBUG { 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "status": "success", 08:23:20 DEBUG "data": { 08:23:20 DEBUG "data": { 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "resultType": "vector", 08:23:20 DEBUG "result": [ 08:23:20 DEBUG "result": [ 08:23:20 DEBUG { 08:23:20 DEBUG { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "metric": { 08:23:20 DEBUG "pod": "ds-cts-2" 08:23:20 DEBUG "instance": "10.142.0.100:9100" 08:23:20 DEBUG }, 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG [...] 08:23:20 DEBUG [...] 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG 08:23:20 DEBUG "pod": "ds-idrepo-1" 08:23:20 DEBUG "instance": "10.142.0.99:9100" 08:23:20 DEBUG }, 08:23:20 DEBUG }, 08:23:20 DEBUG "value": [ 08:23:20 DEBUG "value": [ 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG 1686554600, 08:23:20 DEBUG "0" 08:23:20 DEBUG "6936.366666666667" 08:23:20 DEBUG ] 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 DEBUG ] 08:23:20 DEBUG ] 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 DEBUG } 08:23:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:23:26] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:27] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:34] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:35] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:41] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:42] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:48] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:49] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:23:54 INFO 08:23:54 INFO [loop_until]: kubectl --namespace=xlou top pods 08:23:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:23:54 INFO [loop_until]: OK (rc = 0) 08:23:54 DEBUG --- stdout --- 08:23:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 8m 6125Mi am-7849cf7bdb-npvzp 6m 6234Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 10m 371Mi ds-idrepo-0 16m 13584Mi ds-idrepo-1 18m 13545Mi ds-idrepo-2 18m 13567Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5458Mi idm-58dc667486-vrqsf 6m 5519Mi lodemon-75f677b97d-6b58v 63m 79Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1011m 2646Mi 08:23:54 DEBUG --- stderr --- 08:23:54 DEBUG 127.0.0.1 - - [12/Jun/2023 08:23:56] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:23:56] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:03] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:04] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:24:06 INFO 08:24:06 INFO [loop_until]: kubectl --namespace=xlou top node 08:24:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:24:06 INFO [loop_until]: OK (rc = 0) 08:24:06 DEBUG --- stdout --- 08:24:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 79m 0% 6419Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 7169Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 7059Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 73m 0% 6410Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 66m 0% 7379Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 235m 1% 3044Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 152m 0% 1218Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1122Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1149Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 76m 0% 14537Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 77m 0% 14521Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 68m 0% 1046Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14584Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1076m 6% 3575Mi 6% 08:24:06 DEBUG --- stderr --- 08:24:06 DEBUG 127.0.0.1 - - [12/Jun/2023 08:24:10] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:11] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:18] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:18] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:24:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554659" 08:24:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554659" 08:24:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554659" 08:24:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554659" 08:24:19 INFO [http_cmd]: http status code OK 08:24:19 DEBUG --- status code --- 08:24:19 DEBUG http status code is 200 (expected 200) 08:24:19 DEBUG --- http response --- 08:24:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:19 DEBUG ----- output ----- 08:24:19 DEBUG { 08:24:19 DEBUG "status": "success", 08:24:19 DEBUG "data": { 08:24:19 DEBUG "resultType": "vector", 08:24:19 DEBUG "result": [ 08:24:19 DEBUG { 08:24:19 DEBUG "metric": { 08:24:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG [...] 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 1686554659, 08:24:19 DEBUG "0" 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG } 08:24:19 INFO [http_cmd]: http status code OK 08:24:19 DEBUG --- status code --- 08:24:19 DEBUG http status code is 200 (expected 200) 08:24:19 DEBUG --- http response --- 08:24:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:19 DEBUG ----- output ----- 08:24:19 DEBUG { 08:24:19 DEBUG "status": "success", 08:24:19 DEBUG "data": { 08:24:19 DEBUG "resultType": "vector", 08:24:19 DEBUG "result": [ 08:24:19 DEBUG { 08:24:19 DEBUG "metric": { 08:24:19 DEBUG "instance": "10.142.0.100:9100" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG [...] 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG "instance": "10.142.0.99:9100" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 1686554659, 08:24:19 DEBUG "11091.866666666667" 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG } 08:24:19 INFO [http_cmd]: http status code OK 08:24:19 DEBUG --- status code --- 08:24:19 DEBUG http status code is 200 (expected 200) 08:24:19 DEBUG --- http response --- 08:24:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:19 DEBUG ----- output ----- 08:24:19 DEBUG { 08:24:19 DEBUG "status": "success", 08:24:19 DEBUG "data": { 08:24:19 DEBUG "resultType": "vector", 08:24:19 DEBUG "result": [ 08:24:19 DEBUG { 08:24:19 DEBUG "metric": { 08:24:19 DEBUG "pod": "ds-idrepo-2" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG [...] 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG "pod": "ds-idrepo-1" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 1686554659, 08:24:19 DEBUG "0" 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG } 08:24:19 INFO [http_cmd]: http status code OK 08:24:19 DEBUG --- status code --- 08:24:19 DEBUG http status code is 200 (expected 200) 08:24:19 DEBUG --- http response --- 08:24:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:19 DEBUG ----- output ----- 08:24:19 DEBUG { 08:24:19 DEBUG "status": "success", 08:24:19 DEBUG "data": { 08:24:19 DEBUG "resultType": "vector", 08:24:19 DEBUG "result": [ 08:24:19 DEBUG { 08:24:19 DEBUG "metric": { 08:24:19 DEBUG "pod": "ds-idrepo-2" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG [...] 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG "pod": "ds-idrepo-1" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 1686554659, 08:24:19 DEBUG "0" 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG } 08:24:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554659" 08:24:19 INFO [http_cmd]: http status code OK 08:24:19 DEBUG --- status code --- 08:24:19 DEBUG http status code is 200 (expected 200) 08:24:19 DEBUG --- http response --- 08:24:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:19 DEBUG ----- output ----- 08:24:19 DEBUG { 08:24:19 DEBUG "status": "success", 08:24:19 DEBUG "data": { 08:24:19 DEBUG "resultType": "vector", 08:24:19 DEBUG "result": [ 08:24:19 DEBUG { 08:24:19 DEBUG "metric": { 08:24:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG [...] 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 1686554659, 08:24:19 DEBUG "0" 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG } 08:24:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554659" 08:24:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554659" 08:24:19 INFO [http_cmd]: http status code OK 08:24:19 DEBUG --- status code --- 08:24:19 DEBUG http status code is 200 (expected 200) 08:24:19 DEBUG --- http response --- 08:24:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:24:19 DEBUG ----- output ----- 08:24:19 DEBUG { 08:24:19 DEBUG "status": "success", 08:24:19 DEBUG "data": { 08:24:19 DEBUG "resultType": "vector", 08:24:19 DEBUG "result": [ 08:24:19 DEBUG { 08:24:19 DEBUG "metric": { 08:24:19 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG [...] 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG "pod": "ds-cts-0" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 1686554659, 08:24:19 DEBUG "0" 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG } 08:24:19 INFO [http_cmd]: http status code OK 08:24:19 DEBUG --- status code --- 08:24:19 DEBUG http status code is 200 (expected 200) 08:24:19 DEBUG --- http response --- 08:24:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:24:19 DEBUG ----- output ----- 08:24:19 DEBUG { 08:24:19 DEBUG "status": "success", 08:24:19 DEBUG "data": { 08:24:19 DEBUG "resultType": "vector", 08:24:19 DEBUG "result": [ 08:24:19 DEBUG { 08:24:19 DEBUG "metric": { 08:24:19 DEBUG "pod": "ds-cts-2" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG [...] 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG "pod": "ds-idrepo-1" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 1686554659, 08:24:19 DEBUG "0" 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG } 08:24:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554659" 08:24:19 INFO [http_cmd]: http status code OK 08:24:19 DEBUG --- status code --- 08:24:19 DEBUG http status code is 200 (expected 200) 08:24:19 DEBUG --- http response --- 08:24:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:19 DEBUG ----- output ----- 08:24:19 DEBUG { 08:24:19 DEBUG "status": "success", 08:24:19 DEBUG "data": { 08:24:19 DEBUG "resultType": "vector", 08:24:19 DEBUG "result": [ 08:24:19 DEBUG { 08:24:19 DEBUG "metric": { 08:24:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG [...] 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG 08:24:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:24:19 DEBUG }, 08:24:19 DEBUG "value": [ 08:24:19 DEBUG 1686554659, 08:24:19 DEBUG "0" 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG ] 08:24:19 DEBUG } 08:24:19 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554660" 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "pod": "ds-cts-2" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "pod": "ds-idrepo-1" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "0" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "pod": "ds-idrepo-2" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "pod": "ds-idrepo-1" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "0" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "pod": "ds-cts-0" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "0" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554660" 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "1.0321307640281498" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "instance": "10.142.0.100:9100" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "instance": "10.142.0.99:9100" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "0.0003333333333333262" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554660" 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "0.2" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "instance": "10.142.0.100:9100" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "instance": "10.142.0.99:9100" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "0.00040000000000001514" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "pod": "ds-cts-0" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "0.009672752089521182" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "0" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "14304649216" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "pod": "ds-cts-0" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "4243.973198867168" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "pod": "ds-cts-2" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "pod": "ds-idrepo-1" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "0" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 08:24:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554660" 08:24:20 INFO [http_cmd]: http status code OK 08:24:20 DEBUG --- status code --- 08:24:20 DEBUG http status code is 200 (expected 200) 08:24:20 DEBUG --- http response --- 08:24:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:20 DEBUG ----- output ----- 08:24:20 DEBUG { 08:24:20 DEBUG "status": "success", 08:24:20 DEBUG "data": { 08:24:20 DEBUG "resultType": "vector", 08:24:20 DEBUG "result": [ 08:24:20 DEBUG { 08:24:20 DEBUG "metric": { 08:24:20 DEBUG "instance": "10.142.0.100:9100" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG [...] 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG 08:24:20 DEBUG "instance": "10.142.0.99:9100" 08:24:20 DEBUG }, 08:24:20 DEBUG "value": [ 08:24:20 DEBUG 1686554660, 08:24:20 DEBUG "5078.866666666667" 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG ] 08:24:20 DEBUG } 08:24:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:24:25] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:26] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:32] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:33] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:34] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:40] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:41] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:42] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:49] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:50] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:51] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 08:24:54 INFO 08:24:54 INFO [loop_until]: kubectl --namespace=xlou top pods 08:24:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:24:54 INFO [loop_until]: OK (rc = 0) 08:24:54 DEBUG --- stdout --- 08:24:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 12m 6125Mi am-7849cf7bdb-npvzp 7m 6234Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 11m 417Mi ds-cts-1 8m 451Mi ds-cts-2 10m 371Mi ds-idrepo-0 15m 13584Mi ds-idrepo-1 18m 13544Mi ds-idrepo-2 17m 13560Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 9m 5458Mi idm-58dc667486-vrqsf 5m 5519Mi lodemon-75f677b97d-6b58v 58m 79Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 521m 2759Mi 08:24:54 DEBUG --- stderr --- 08:24:54 DEBUG 127.0.0.1 - - [12/Jun/2023 08:24:57] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:58] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:24:59] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:05] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:06] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:25:06 INFO 08:25:06 INFO [loop_until]: kubectl --namespace=xlou top node 08:25:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:25:06 INFO [loop_until]: OK (rc = 0) 08:25:06 DEBUG --- stdout --- 08:25:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 66m 0% 6416Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 7168Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 71m 0% 7057Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 73m 0% 6413Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 67m 0% 7378Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 225m 1% 3047Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 151m 0% 1221Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1120Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1151Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 76m 0% 14529Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 80m 0% 14522Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 68m 0% 1047Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14582Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 688m 4% 3658Mi 6% 08:25:06 DEBUG --- stderr --- 08:25:06 DEBUG 127.0.0.1 - - [12/Jun/2023 08:25:07] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:14] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:15] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:16] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 08:25:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554719" 08:25:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554719" 08:25:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554719" 08:25:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554719" 08:25:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554719" 08:25:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554719" 08:25:19 INFO [http_cmd]: http status code OK 08:25:19 INFO [http_cmd]: http status code OK 08:25:19 DEBUG --- status code --- 08:25:19 INFO [http_cmd]: http status code OK 08:25:19 INFO [http_cmd]: http status code OK 08:25:19 DEBUG --- status code --- 08:25:19 DEBUG http status code is 200 (expected 200) 08:25:19 DEBUG --- status code --- 08:25:19 DEBUG --- status code --- 08:25:19 DEBUG http status code is 200 (expected 200) 08:25:19 DEBUG --- http response --- 08:25:19 DEBUG http status code is 200 (expected 200) 08:25:19 DEBUG http status code is 200 (expected 200) 08:25:19 DEBUG --- http response --- 08:25:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:19 DEBUG --- http response --- 08:25:19 DEBUG --- http response --- 08:25:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:19 DEBUG ----- output ----- 08:25:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:25:19 DEBUG ----- output ----- 08:25:19 DEBUG { 08:25:19 DEBUG ----- output ----- 08:25:19 DEBUG ----- output ----- 08:25:19 DEBUG { 08:25:19 DEBUG "status": "success", 08:25:19 DEBUG { 08:25:19 DEBUG { 08:25:19 DEBUG "status": "success", 08:25:19 DEBUG "data": { 08:25:19 DEBUG "status": "success", 08:25:19 DEBUG "status": "success", 08:25:19 DEBUG "data": { 08:25:19 DEBUG "resultType": "vector", 08:25:19 DEBUG "data": { 08:25:19 DEBUG "data": { 08:25:19 DEBUG "resultType": "vector", 08:25:19 DEBUG "result": [ 08:25:19 DEBUG "resultType": "vector", 08:25:19 DEBUG "resultType": "vector", 08:25:19 DEBUG "result": [ 08:25:19 DEBUG { 08:25:19 DEBUG "result": [ 08:25:19 DEBUG "result": [ 08:25:19 DEBUG { 08:25:19 DEBUG "metric": { 08:25:19 DEBUG { 08:25:19 DEBUG { 08:25:19 DEBUG "metric": { 08:25:19 DEBUG "pod": "ds-idrepo-2" 08:25:19 DEBUG "metric": { 08:25:19 DEBUG "metric": { 08:25:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:25:19 DEBUG }, 08:25:19 DEBUG "pod": "ds-idrepo-2" 08:25:19 DEBUG "pod": "ds-cts-2" 08:25:19 DEBUG }, 08:25:19 DEBUG "value": [ 08:25:19 DEBUG }, 08:25:19 DEBUG }, 08:25:19 DEBUG "value": [ 08:25:19 DEBUG 08:25:19 DEBUG "value": [ 08:25:19 DEBUG "value": [ 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG [...] 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG [...] 08:25:19 DEBUG 08:25:19 DEBUG [...] 08:25:19 DEBUG [...] 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG "pod": "ds-idrepo-1" 08:25:19 DEBUG }, 08:25:19 DEBUG "pod": "ds-idrepo-1" 08:25:19 DEBUG "pod": "ds-idrepo-1" 08:25:19 DEBUG }, 08:25:19 DEBUG "value": [ 08:25:19 DEBUG }, 08:25:19 DEBUG }, 08:25:19 DEBUG "value": [ 08:25:19 DEBUG 1686554719, 08:25:19 DEBUG "value": [ 08:25:19 DEBUG "value": [ 08:25:19 DEBUG 1686554719, 08:25:19 DEBUG "0" 08:25:19 INFO [http_cmd]: http status code OK 08:25:19 DEBUG 1686554719, 08:25:19 INFO [http_cmd]: http status code OK 08:25:19 DEBUG 1686554719, 08:25:19 DEBUG "0" 08:25:19 DEBUG ] 08:25:19 DEBUG --- status code --- 08:25:19 DEBUG "0" 08:25:19 DEBUG --- status code --- 08:25:19 DEBUG "0" 08:25:19 DEBUG ] 08:25:19 DEBUG } 08:25:19 DEBUG http status code is 200 (expected 200) 08:25:19 DEBUG ] 08:25:19 DEBUG http status code is 200 (expected 200) 08:25:19 DEBUG ] 08:25:19 DEBUG } 08:25:19 DEBUG ] 08:25:19 DEBUG --- http response --- 08:25:19 DEBUG } 08:25:19 DEBUG --- http response --- 08:25:19 DEBUG } 08:25:19 DEBUG ] 08:25:19 DEBUG } 08:25:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:19 DEBUG ] 08:25:19 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:19 DEBUG ] 08:25:19 DEBUG } 08:25:19 DEBUG } 08:25:19 DEBUG ----- output ----- 08:25:19 DEBUG } 08:25:19 DEBUG ----- output ----- 08:25:19 DEBUG } 08:25:19 DEBUG } 08:25:19 DEBUG { 08:25:19 DEBUG } 08:25:19 DEBUG { 08:25:19 DEBUG } 08:25:19 DEBUG "status": "success", 08:25:19 DEBUG "status": "success", 08:25:19 DEBUG "data": { 08:25:19 DEBUG "data": { 08:25:19 DEBUG "resultType": "vector", 08:25:19 DEBUG "resultType": "vector", 08:25:19 DEBUG "result": [ 08:25:19 DEBUG "result": [ 08:25:19 DEBUG { 08:25:19 DEBUG { 08:25:19 DEBUG "metric": { 08:25:19 DEBUG "metric": { 08:25:19 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:25:19 DEBUG "instance": "10.142.0.100:9100" 08:25:19 DEBUG }, 08:25:19 DEBUG }, 08:25:19 DEBUG "value": [ 08:25:19 DEBUG "value": [ 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG [...] 08:25:19 DEBUG [...] 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG 08:25:19 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:25:19 DEBUG "instance": "10.142.0.99:9100" 08:25:19 DEBUG }, 08:25:19 DEBUG }, 08:25:19 DEBUG "value": [ 08:25:19 DEBUG "value": [ 08:25:19 DEBUG 1686554719, 08:25:19 DEBUG 1686554719, 08:25:19 DEBUG "0" 08:25:19 DEBUG "10248.133333333333" 08:25:19 DEBUG ] 08:25:19 DEBUG ] 08:25:19 DEBUG } 08:25:19 DEBUG } 08:25:19 DEBUG ] 08:25:19 DEBUG ] 08:25:19 DEBUG } 08:25:19 DEBUG } 08:25:19 DEBUG } 08:25:19 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "ds-idrepo-2" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "ds-idrepo-1" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "ds-cts-2" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "ds-idrepo-1" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "instance": "10.142.0.100:9100" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "instance": "10.142.0.99:9100" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0.9670326413859124" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0.2" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "instance": "10.142.0.100:9100" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "instance": "10.142.0.99:9100" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0.0003333333333330302" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0.006078927840754047" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554720" 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "14297530368" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "pod": "ds-cts-2" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "pod": "ds-idrepo-1" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "0" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 08:25:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554720" 08:25:20 INFO [http_cmd]: http status code OK 08:25:20 DEBUG --- status code --- 08:25:20 DEBUG http status code is 200 (expected 200) 08:25:20 DEBUG --- http response --- 08:25:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:20 DEBUG ----- output ----- 08:25:20 DEBUG { 08:25:20 DEBUG "status": "success", 08:25:20 DEBUG "data": { 08:25:20 DEBUG "resultType": "vector", 08:25:20 DEBUG "result": [ 08:25:20 DEBUG { 08:25:20 DEBUG "metric": { 08:25:20 DEBUG "instance": "10.142.0.100:9100" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG [...] 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG 08:25:20 DEBUG "instance": "10.142.0.99:9100" 08:25:20 DEBUG }, 08:25:20 DEBUG "value": [ 08:25:20 DEBUG 1686554720, 08:25:20 DEBUG "4382.433333333333" 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG ] 08:25:20 DEBUG } 08:25:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:25:22] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:23] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:24] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:30] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:31] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:32] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:38] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:39] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:40] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:47] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:48] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:49] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 08:25:54 INFO 08:25:54 INFO [loop_until]: kubectl --namespace=xlou top pods 08:25:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:25:54 INFO [loop_until]: OK (rc = 0) 08:25:54 DEBUG --- stdout --- 08:25:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 9m 6125Mi am-7849cf7bdb-npvzp 7m 6234Mi am-7849cf7bdb-p4fns 9m 6287Mi ds-cts-0 10m 416Mi ds-cts-1 9m 451Mi ds-cts-2 10m 371Mi ds-idrepo-0 100m 13585Mi ds-idrepo-1 18m 13544Mi ds-idrepo-2 17m 13560Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5458Mi idm-58dc667486-vrqsf 5m 5519Mi lodemon-75f677b97d-6b58v 83m 79Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1015m 2990Mi 08:25:54 DEBUG --- stderr --- 08:25:54 DEBUG 127.0.0.1 - - [12/Jun/2023 08:25:55] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:56] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:25:57] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:03] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:04] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:05] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 08:26:07 INFO 08:26:07 INFO [loop_until]: kubectl --namespace=xlou top node 08:26:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:26:07 INFO [loop_until]: OK (rc = 0) 08:26:07 DEBUG --- stdout --- 08:26:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 64m 0% 6414Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 7165Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 66m 0% 7054Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 73m 0% 6415Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 65m 0% 7378Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 223m 1% 3034Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 179m 1% 1218Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1121Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1152Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 90m 0% 14532Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 79m 0% 14524Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 68m 0% 1051Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 177m 1% 14583Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1077m 6% 3889Mi 6% 08:26:07 DEBUG --- stderr --- 08:26:07 DEBUG 127.0.0.1 - - [12/Jun/2023 08:26:11] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:12] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:13] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 08:26:19 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554779" 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "pod": "ds-cts-2" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "ds-idrepo-1" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554779, 08:26:20 DEBUG "0" 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "pod": "ds-idrepo-2" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "ds-idrepo-1" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "0" 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554780" 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "0" 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG { 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "result": [ 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 DEBUG "metric": { 08:26:20 DEBUG { 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "pod": "ds-idrepo-2" 08:26:20 DEBUG }, 08:26:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:26:20 DEBUG "instance": "10.142.0.100:9100" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG }, 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG "value": [ 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "ds-idrepo-1" 08:26:20 DEBUG }, 08:26:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:26:20 DEBUG "instance": "10.142.0.99:9100" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG }, 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "0" 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "0" 08:26:20 DEBUG ] 08:26:20 DEBUG "0" 08:26:20 DEBUG "11116.433333333332" 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "0" 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "pod": "ds-cts-2" 08:26:20 DEBUG "pod": "ds-idrepo-2" 08:26:20 DEBUG }, 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "ds-idrepo-1" 08:26:20 DEBUG "pod": "ds-idrepo-1" 08:26:20 DEBUG }, 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "0" 08:26:20 DEBUG "0" 08:26:20 DEBUG ] 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554780" 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554780" 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 DEBUG --- status code --- 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554780" 127.0.0.1 - - [12/Jun/2023 08:26:20] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 DEBUG --- http response --- 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG { 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG "data": { 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG { 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG }, 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "value": [ 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "instance": "10.142.0.100:9100" 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG 08:26:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:26:20 DEBUG }, 08:26:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:26:20 DEBUG }, 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG { 08:26:20 DEBUG { 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "data": { 08:26:20 DEBUG "data": { 08:26:20 DEBUG }, 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:26:20 DEBUG 08:26:20 DEBUG "instance": "10.142.0.99:9100" 08:26:20 DEBUG "result": [ 08:26:20 DEBUG "result": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:26:20 DEBUG }, 08:26:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:26:20 DEBUG }, 08:26:20 DEBUG { 08:26:20 DEBUG { 08:26:20 DEBUG "0" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG ] 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:26:20 DEBUG "instance": "10.142.0.100:9100" 08:26:20 DEBUG } 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "1.0354731914089979" 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "0" 08:26:20 DEBUG }, 08:26:20 DEBUG }, 08:26:20 DEBUG ] 08:26:20 DEBUG "2.009858437737777" 08:26:20 DEBUG ] 08:26:20 DEBUG "0.2" 08:26:20 DEBUG ] 08:26:20 DEBUG "value": [ 08:26:20 DEBUG "value": [ 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG [...] 08:26:20 DEBUG [...] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:26:20 DEBUG "instance": "10.142.0.99:9100" 08:26:20 DEBUG }, 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "0" 08:26:20 DEBUG "0.00030000000000048505" 08:26:20 DEBUG ] 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554780" 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "14297993216" 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "536328.3584589285" 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554780" 08:26:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554780" 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "pod": "ds-cts-2" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "pod": "ds-idrepo-1" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "0" 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 08:26:20 INFO [http_cmd]: http status code OK 08:26:20 DEBUG --- status code --- 08:26:20 DEBUG http status code is 200 (expected 200) 08:26:20 DEBUG --- http response --- 08:26:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:20 DEBUG ----- output ----- 08:26:20 DEBUG { 08:26:20 DEBUG "status": "success", 08:26:20 DEBUG "data": { 08:26:20 DEBUG "resultType": "vector", 08:26:20 DEBUG "result": [ 08:26:20 DEBUG { 08:26:20 DEBUG "metric": { 08:26:20 DEBUG "instance": "10.142.0.100:9100" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG [...] 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG 08:26:20 DEBUG "instance": "10.142.0.99:9100" 08:26:20 DEBUG }, 08:26:20 DEBUG "value": [ 08:26:20 DEBUG 1686554780, 08:26:20 DEBUG "4872.4" 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG ] 08:26:20 DEBUG } 08:26:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:26:21] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:22] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:28] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:29] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:30] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:36] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:37] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:38] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:45] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:46] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:47] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:26:53] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:26:54 INFO 08:26:54 INFO [loop_until]: kubectl --namespace=xlou top pods 08:26:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 127.0.0.1 - - [12/Jun/2023 08:26:54] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:26:54 INFO [loop_until]: OK (rc = 0) 08:26:54 DEBUG --- stdout --- 08:26:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 8m 6125Mi am-7849cf7bdb-npvzp 6m 6234Mi am-7849cf7bdb-p4fns 7m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 16m 13584Mi ds-idrepo-1 18m 13544Mi ds-idrepo-2 16m 13560Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 8m 5458Mi idm-58dc667486-vrqsf 6m 5519Mi lodemon-75f677b97d-6b58v 78m 79Mi login-ui-55f8bc5458-nxdzc 1m 4Mi overseer-0-5b849496f7-qmtsz 1013m 3166Mi 08:26:54 DEBUG --- stderr --- 08:26:54 DEBUG 127.0.0.1 - - [12/Jun/2023 08:26:55] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:01] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:02] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:03] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 08:27:07 INFO 08:27:07 INFO [loop_until]: kubectl --namespace=xlou top node 08:27:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:27:07 INFO [loop_until]: OK (rc = 0) 08:27:07 DEBUG --- stdout --- 08:27:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 68m 0% 6415Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 7166Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 7054Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 74m 0% 6413Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 66m 0% 7381Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 227m 1% 3052Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 161m 1% 1221Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 67m 0% 1117Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1151Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 76m 0% 14531Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 80m 0% 14524Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 69m 0% 1051Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14588Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1033m 6% 4075Mi 6% 08:27:07 DEBUG --- stderr --- 08:27:07 DEBUG 127.0.0.1 - - [12/Jun/2023 08:27:10] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:11] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:11] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:18] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:19] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "pod": "ds-cts-2" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "pod": "ds-idrepo-1" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "0" 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "pod": "ds-idrepo-2" 08:27:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:27:20 DEBUG }, 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "pod": "ds-idrepo-1" 08:27:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:27:20 DEBUG }, 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "0" 08:27:20 DEBUG "0" 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 INFO [http_cmd]: http status code OK 127.0.0.1 - - [12/Jun/2023 08:27:20] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 08:27:20 DEBUG --- status code --- 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG { 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG { 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "pod": "ds-idrepo-2" 08:27:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:27:20 DEBUG }, 08:27:20 DEBUG "instance": "10.142.0.100:9100" 08:27:20 DEBUG "pod": "ds-idrepo-2" 08:27:20 DEBUG }, 08:27:20 DEBUG "pod": "ds-cts-2" 08:27:20 DEBUG }, 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:27:20 DEBUG 08:27:20 DEBUG "pod": "ds-idrepo-1" 08:27:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:27:20 DEBUG }, 08:27:20 DEBUG "instance": "10.142.0.99:9100" 08:27:20 DEBUG "pod": "ds-idrepo-1" 08:27:20 DEBUG }, 08:27:20 DEBUG "pod": "ds-idrepo-1" 08:27:20 DEBUG }, 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "0" 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "0" 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "0" 08:27:20 DEBUG "0" 08:27:20 DEBUG ] 08:27:20 DEBUG "11269.233333333334" 08:27:20 DEBUG "0" 08:27:20 DEBUG ] 08:27:20 DEBUG "0" 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554840" 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "pod": "ds-cts-0" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "0" 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "1.059624258938063" 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554840" 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:20 DEBUG { 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG { 08:27:20 DEBUG { 08:27:20 DEBUG "data": { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "data": { 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG { 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "metric": { 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG { 08:27:20 DEBUG { 08:27:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "instance": "10.142.0.100:9100" 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG }, 08:27:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:27:20 DEBUG }, 08:27:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:27:20 DEBUG "instance": "10.142.0.100:9100" 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG }, 08:27:20 DEBUG 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:27:20 DEBUG 08:27:20 DEBUG "instance": "10.142.0.99:9100" 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG }, 08:27:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:27:20 DEBUG }, 08:27:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:27:20 DEBUG "instance": "10.142.0.99:9100" 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG }, 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "0" 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "0.0000999999999995301" 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG ] 08:27:20 DEBUG "0.2" 08:27:20 DEBUG ] 08:27:20 DEBUG "0" 08:27:20 DEBUG "0" 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:27:20 DEBUG { 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG { 08:27:20 DEBUG "data": { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG { 08:27:20 DEBUG "result": [ 08:27:20 DEBUG "metric": { 08:27:20 DEBUG { 08:27:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:27:20 DEBUG "metric": { 08:27:20 DEBUG }, 08:27:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:27:20 DEBUG 08:27:20 DEBUG }, 08:27:20 DEBUG "pod": "ds-cts-0" 08:27:20 DEBUG "value": [ 08:27:20 DEBUG }, 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG "14296764416" 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG ] 08:27:20 DEBUG "0" 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554840" 08:27:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554840" 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "pod": "ds-cts-2" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "pod": "ds-idrepo-1" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "0" 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 08:27:20 INFO [http_cmd]: http status code OK 08:27:20 DEBUG --- status code --- 08:27:20 DEBUG http status code is 200 (expected 200) 08:27:20 DEBUG --- http response --- 08:27:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:20 DEBUG ----- output ----- 08:27:20 DEBUG { 08:27:20 DEBUG "status": "success", 08:27:20 DEBUG "data": { 08:27:20 DEBUG "resultType": "vector", 08:27:20 DEBUG "result": [ 08:27:20 DEBUG { 08:27:20 DEBUG "metric": { 08:27:20 DEBUG "instance": "10.142.0.100:9100" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG [...] 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG 08:27:20 DEBUG "instance": "10.142.0.99:9100" 08:27:20 DEBUG }, 08:27:20 DEBUG "value": [ 08:27:20 DEBUG 1686554840, 08:27:20 DEBUG "4928.599999999999" 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG ] 08:27:20 DEBUG } 08:27:20 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:27:26] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:27] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:28] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:34] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:35] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:36] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:37] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:44] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:45] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:46] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:47] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:53] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:54] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:27:54 INFO 08:27:54 INFO [loop_until]: kubectl --namespace=xlou top pods 08:27:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:27:54 INFO [loop_until]: OK (rc = 0) 08:27:54 DEBUG --- stdout --- 08:27:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 10m 6125Mi am-7849cf7bdb-npvzp 8m 6234Mi am-7849cf7bdb-p4fns 8m 6287Mi ds-cts-0 10m 417Mi ds-cts-1 9m 451Mi ds-cts-2 10m 370Mi ds-idrepo-0 16m 13584Mi ds-idrepo-1 18m 13544Mi ds-idrepo-2 18m 13560Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 3m 5458Mi idm-58dc667486-vrqsf 6m 5519Mi lodemon-75f677b97d-6b58v 78m 79Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 175m 3113Mi 08:27:54 DEBUG --- stderr --- 08:27:54 DEBUG 127.0.0.1 - - [12/Jun/2023 08:27:55] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:27:56] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:02] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:03] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:04] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:05] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 08:28:07 INFO 08:28:07 INFO [loop_until]: kubectl --namespace=xlou top node 08:28:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:28:07 INFO [loop_until]: OK (rc = 0) 08:28:07 DEBUG --- stdout --- 08:28:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 68m 0% 6417Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 7168Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 7052Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 68m 0% 6413Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 67m 0% 7381Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 229m 1% 3048Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 169m 1% 1219Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 72m 0% 1120Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1149Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 77m 0% 14528Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 81m 0% 14526Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 70m 0% 1052Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14585Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 473m 2% 3996Mi 6% 08:28:07 DEBUG --- stderr --- 08:28:07 DEBUG 127.0.0.1 - - [12/Jun/2023 08:28:11] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:12] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:13] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:14] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "result": [ 08:28:20 DEBUG { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "pod": "ds-cts-2" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG [...] 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "pod": "ds-idrepo-1" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "0" 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "result": [ 08:28:20 DEBUG { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "pod": "ds-idrepo-2" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG [...] 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "pod": "ds-idrepo-1" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "0" 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "result": [ 08:28:20 DEBUG { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG [...] 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "0" 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554900" 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "result": [ 08:28:20 DEBUG { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [...] 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:20 DEBUG 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG { 08:28:20 DEBUG 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG }, 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "value": [ 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "result": [ 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG "result": [ 08:28:20 DEBUG { 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG "0" 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG --- status code --- 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 DEBUG ] 08:28:20 DEBUG --- status code --- 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554900" 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG } 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:20 DEBUG "instance": "10.142.0.100:9100" 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:20 DEBUG }, 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG ] 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG }, 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG "value": [ 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:20 DEBUG } 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:28:20 DEBUG { 08:28:20 DEBUG "value": [ 08:28:20 DEBUG { 08:28:20 DEBUG 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG } 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG 08:28:20 DEBUG { 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG 08:28:20 DEBUG "data": { 08:28:20 DEBUG 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG "data": { 08:28:20 DEBUG 08:28:20 DEBUG "resultType": "vector", 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG [...] 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG [...] 08:28:20 DEBUG "result": [ 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "result": [ 08:28:20 DEBUG "result": [ 08:28:20 DEBUG 08:28:20 DEBUG { 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG 08:28:20 DEBUG { 08:28:20 DEBUG { 08:28:20 DEBUG 08:28:20 DEBUG "metric": { 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG "result": [ 08:28:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG 08:28:20 DEBUG "pod": "ds-idrepo-2" 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:28:20 DEBUG { 08:28:20 DEBUG }, 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG "pod": "ds-idrepo-2" 08:28:20 DEBUG "pod": "ds-cts-2" 08:28:20 DEBUG "instance": "10.142.0.99:9100" 08:28:20 DEBUG }, 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "value": [ 08:28:20 DEBUG --- status code --- 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG }, 08:28:20 DEBUG }, 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG { 08:28:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG "value": [ 08:28:20 DEBUG "value": [ 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG }, 08:28:20 DEBUG "0" 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG 08:28:20 DEBUG "data": { 08:28:20 DEBUG "value": [ 08:28:20 DEBUG ] 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "11471.1" 08:28:20 DEBUG 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG 08:28:20 DEBUG } 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG ] 08:28:20 DEBUG [...] 08:28:20 DEBUG "result": [ 08:28:20 DEBUG 08:28:20 DEBUG ] 08:28:20 DEBUG { 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG [...] 08:28:20 DEBUG [...] 08:28:20 DEBUG } 08:28:20 DEBUG 08:28:20 DEBUG { 08:28:20 DEBUG 08:28:20 DEBUG } 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG { 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG ] 08:28:20 DEBUG 08:28:20 DEBUG "metric": { 08:28:20 DEBUG [...] 08:28:20 DEBUG } 08:28:20 DEBUG "data": { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG } 08:28:20 DEBUG 08:28:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:28:20 DEBUG 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "data": { 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG } 08:28:20 DEBUG "pod": "ds-idrepo-1" 08:28:20 DEBUG }, 08:28:20 DEBUG 08:28:20 DEBUG "result": [ 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "pod": "ds-idrepo-1" 08:28:20 DEBUG "pod": "ds-idrepo-1" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG { 08:28:20 DEBUG "result": [ 08:28:20 DEBUG }, 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:28:20 DEBUG "metric": { 08:28:20 DEBUG { 08:28:20 DEBUG "value": [ 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG 08:28:20 DEBUG }, 08:28:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:28:20 DEBUG "metric": { 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "0" 08:28:20 DEBUG 08:28:20 DEBUG "value": [ 08:28:20 DEBUG }, 08:28:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:28:20 DEBUG "0" 08:28:20 DEBUG "0" 08:28:20 DEBUG ] 08:28:20 DEBUG [...] 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG }, 08:28:20 DEBUG ] 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG 08:28:20 DEBUG "0" 08:28:20 DEBUG 08:28:20 DEBUG "value": [ 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG 08:28:20 DEBUG ] 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG ] 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG 08:28:20 DEBUG } 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:28:20 DEBUG ] 08:28:20 DEBUG [...] 08:28:20 DEBUG 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG }, 08:28:20 DEBUG } 08:28:20 DEBUG 08:28:20 DEBUG [...] 08:28:20 DEBUG "value": [ 08:28:20 DEBUG } 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "0" 08:28:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:28:20 DEBUG 08:28:20 DEBUG ] 08:28:20 DEBUG }, 08:28:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:28:20 DEBUG } 08:28:20 DEBUG "value": [ 08:28:20 DEBUG }, 08:28:20 DEBUG ] 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG } 08:28:20 DEBUG "0.3273368973321937" 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG "1.2626103404441953" 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554900" 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "result": [ 08:28:20 DEBUG { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "instance": "10.142.0.100:9100" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG [...] 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "instance": "10.142.0.99:9100" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "0" 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554900" 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG { 08:28:20 DEBUG { 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG "data": { 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "result": [ 08:28:20 DEBUG "result": [ 08:28:20 DEBUG "result": [ 08:28:20 DEBUG { 08:28:20 DEBUG { 08:28:20 DEBUG { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:28:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:28:20 DEBUG "instance": "10.142.0.100:9100" 08:28:20 DEBUG }, 08:28:20 DEBUG }, 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG "value": [ 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG [...] 08:28:20 DEBUG [...] 08:28:20 DEBUG [...] 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:28:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:28:20 DEBUG "instance": "10.142.0.99:9100" 08:28:20 DEBUG }, 08:28:20 DEBUG }, 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG "value": [ 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "0.2" 08:28:20 DEBUG "0" 08:28:20 DEBUG "0.0004333333333325603" 08:28:20 DEBUG ] 08:28:20 DEBUG ] 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG ] 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554900" 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "result": [ 08:28:20 DEBUG { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG [...] 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "14296616960" 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554900" 08:28:20 INFO [http_cmd]: http status code OK 08:28:20 DEBUG --- status code --- 08:28:20 DEBUG http status code is 200 (expected 200) 08:28:20 DEBUG --- http response --- 08:28:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:28:20 DEBUG ----- output ----- 08:28:20 DEBUG { 08:28:20 DEBUG "status": "success", 08:28:20 DEBUG "data": { 08:28:20 DEBUG "resultType": "vector", 08:28:20 DEBUG "result": [ 08:28:20 DEBUG { 08:28:20 DEBUG "metric": { 08:28:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG [...] 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG 08:28:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:28:20 DEBUG }, 08:28:20 DEBUG "value": [ 08:28:20 DEBUG 1686554900, 08:28:20 DEBUG "0" 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG ] 08:28:20 DEBUG } 08:28:20 DEBUG } 08:28:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554901" 08:28:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554901" 127.0.0.1 - - [12/Jun/2023 08:28:21] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:28:21 INFO [http_cmd]: http status code OK 08:28:21 DEBUG --- status code --- 08:28:21 DEBUG http status code is 200 (expected 200) 08:28:21 DEBUG --- http response --- 08:28:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:21 DEBUG ----- output ----- 08:28:21 DEBUG { 08:28:21 DEBUG "status": "success", 08:28:21 DEBUG "data": { 08:28:21 DEBUG "resultType": "vector", 08:28:21 DEBUG "result": [ 08:28:21 DEBUG { 08:28:21 DEBUG "metric": { 08:28:21 DEBUG "instance": "10.142.0.100:9100" 08:28:21 INFO [http_cmd]: http status code OK 08:28:21 DEBUG }, 08:28:21 DEBUG --- status code --- 08:28:21 DEBUG "value": [ 08:28:21 DEBUG http status code is 200 (expected 200) 08:28:21 DEBUG 08:28:21 DEBUG --- http response --- 08:28:21 DEBUG 08:28:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:28:21 DEBUG 08:28:21 DEBUG ----- output ----- 08:28:21 DEBUG [...] 08:28:21 DEBUG { 08:28:21 DEBUG 08:28:21 DEBUG "status": "success", 08:28:21 DEBUG 08:28:21 DEBUG "data": { 08:28:21 DEBUG 08:28:21 DEBUG "resultType": "vector", 08:28:21 DEBUG "instance": "10.142.0.99:9100" 08:28:21 DEBUG "result": [ 08:28:21 DEBUG }, 08:28:21 DEBUG { 08:28:21 DEBUG "value": [ 08:28:21 DEBUG "metric": { 08:28:21 DEBUG 1686554901, 08:28:21 DEBUG "pod": "ds-cts-2" 08:28:21 DEBUG "7246" 08:28:21 DEBUG }, 08:28:21 DEBUG ] 08:28:21 DEBUG "value": [ 08:28:21 DEBUG } 08:28:21 DEBUG 08:28:21 DEBUG ] 08:28:21 DEBUG 08:28:21 DEBUG } 08:28:21 DEBUG 08:28:21 DEBUG } 08:28:21 DEBUG [...] 08:28:21 DEBUG 08:28:21 DEBUG 08:28:21 DEBUG 08:28:21 DEBUG "pod": "ds-idrepo-1" 08:28:21 DEBUG }, 08:28:21 DEBUG "value": [ 08:28:21 DEBUG 1686554901, 08:28:21 DEBUG "0" 08:28:21 DEBUG ] 08:28:21 DEBUG } 08:28:21 DEBUG ] 08:28:21 DEBUG } 08:28:21 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:28:22] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:22] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:23] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:30] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:31] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:32] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:33] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:39] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:40] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:41] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:42] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:48] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:49] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:50] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:51] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 08:28:54 INFO 08:28:54 INFO [loop_until]: kubectl --namespace=xlou top pods 08:28:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:28:54 INFO [loop_until]: OK (rc = 0) 08:28:54 DEBUG --- stdout --- 08:28:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 9m 6125Mi am-7849cf7bdb-npvzp 7m 6234Mi am-7849cf7bdb-p4fns 9m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 9m 370Mi ds-idrepo-0 16m 13585Mi ds-idrepo-1 18m 13544Mi ds-idrepo-2 18m 13560Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 4m 5457Mi idm-58dc667486-vrqsf 6m 5518Mi lodemon-75f677b97d-6b58v 102m 79Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1000m 3528Mi 08:28:54 DEBUG --- stderr --- 08:28:54 DEBUG 127.0.0.1 - - [12/Jun/2023 08:28:57] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:58] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:28:59] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:00] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:07] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 08:29:07 INFO 08:29:07 INFO [loop_until]: kubectl --namespace=xlou top node 08:29:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:29:07 INFO [loop_until]: OK (rc = 0) 08:29:07 DEBUG --- stdout --- 08:29:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 66m 0% 6412Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 7164Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 66m 0% 7058Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 69m 0% 6414Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 68m 0% 7373Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 220m 1% 3033Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 187m 1% 1221Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 70m 0% 1117Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 68m 0% 1151Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 74m 0% 14531Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 75m 0% 14526Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 68m 0% 1051Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 69m 0% 14586Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1077m 6% 4394Mi 7% 08:29:07 DEBUG --- stderr --- 08:29:07 DEBUG 127.0.0.1 - - [12/Jun/2023 08:29:08] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:09] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:10] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:16] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:17] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:18] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:19] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG "result": [ 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG { 08:29:20 DEBUG { 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG "pod": "ds-idrepo-2" 08:29:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG }, 08:29:20 DEBUG }, 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "value": [ 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG { 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "data": { 08:29:20 DEBUG [...] 08:29:20 DEBUG [...] 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "result": [ 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG { 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "pod": "ds-idrepo-1" 08:29:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:29:20 DEBUG "pod": "ds-cts-2" 08:29:20 DEBUG }, 08:29:20 DEBUG }, 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG 08:29:20 DEBUG "0" 08:29:20 DEBUG "0" 08:29:20 DEBUG 08:29:20 DEBUG ] 08:29:20 DEBUG ] 08:29:20 DEBUG 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG [...] 08:29:20 DEBUG ] 08:29:20 DEBUG ] 08:29:20 DEBUG 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG 08:29:20 DEBUG "pod": "ds-idrepo-1" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "0" 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG [...] 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "0" 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "pod": "ds-cts-2" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG [...] 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "pod": "ds-idrepo-1" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "0" 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG [...] 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "0" 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG [...] 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "0" 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686554960" 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG "data": { 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG "result": [ 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG { 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:20 DEBUG "metric": { 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG "instance": "10.142.0.100:9100" 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG { 08:29:20 DEBUG }, 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG "data": { 08:29:20 DEBUG 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:29:20 DEBUG "result": [ 08:29:20 DEBUG 08:29:20 DEBUG "result": [ 08:29:20 DEBUG { 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG [...] 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG { 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "pod": "ds-idrepo-2" 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "pod": "ds-idrepo-2" 08:29:20 DEBUG 08:29:20 DEBUG "instance": "10.142.0.100:9100" 08:29:20 DEBUG }, 08:29:20 DEBUG "data": { 08:29:20 DEBUG "data": { 08:29:20 DEBUG }, 08:29:20 DEBUG 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "instance": "10.142.0.99:9100" 08:29:20 DEBUG "value": [ 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG 08:29:20 DEBUG "result": [ 08:29:20 DEBUG "result": [ 08:29:20 DEBUG 08:29:20 DEBUG }, 08:29:20 DEBUG 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG 08:29:20 DEBUG { 08:29:20 DEBUG { 08:29:20 DEBUG 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [...] 08:29:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:29:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:29:20 DEBUG [...] 08:29:20 DEBUG "11534.266666666666" 08:29:20 DEBUG [...] 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:20 DEBUG 08:29:20 DEBUG }, 08:29:20 DEBUG }, 08:29:20 DEBUG 08:29:20 DEBUG ] 08:29:20 DEBUG 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG } 08:29:20 DEBUG 08:29:20 DEBUG { 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG ] 08:29:20 DEBUG 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "pod": "ds-idrepo-1" 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "pod": "ds-idrepo-1" 08:29:20 DEBUG } 08:29:20 DEBUG "instance": "10.142.0.99:9100" 08:29:20 DEBUG "data": { 08:29:20 DEBUG }, 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG }, 08:29:20 DEBUG } 08:29:20 DEBUG }, 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "value": [ 08:29:20 DEBUG [...] 08:29:20 DEBUG [...] 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "result": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG { 08:29:20 DEBUG "0" 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "0" 08:29:20 DEBUG "0.00036666666666747006" 08:29:20 DEBUG "metric": { 08:29:20 DEBUG ] 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG ] 08:29:20 DEBUG ] 08:29:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:29:20 DEBUG } 08:29:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:29:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG }, 08:29:20 DEBUG ] 08:29:20 DEBUG }, 08:29:20 DEBUG }, 08:29:20 DEBUG ] 08:29:20 DEBUG ] 08:29:20 DEBUG "value": [ 08:29:20 DEBUG } 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "value": [ 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG 08:29:20 DEBUG } 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG 08:29:20 DEBUG "0.7393041759271226" 08:29:20 DEBUG "0" 08:29:20 DEBUG 08:29:20 DEBUG ] 08:29:20 DEBUG ] 08:29:20 DEBUG [...] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG 08:29:20 DEBUG ] 08:29:20 DEBUG ] 08:29:20 DEBUG 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "1.0392913535390658" 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554960" 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "instance": "10.142.0.100:9100" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG [...] 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "instance": "10.142.0.99:9100" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "0" 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG "result": [ 08:29:20 DEBUG { 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:29:20 DEBUG "pod": "am-7849cf7bdb-bd9j2" 08:29:20 DEBUG }, 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG [...] 08:29:20 DEBUG [...] 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:29:20 DEBUG "pod": "am-7849cf7bdb-npvzp" 08:29:20 DEBUG }, 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "0.2" 08:29:20 DEBUG "0" 08:29:20 DEBUG ] 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686554960" 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554960" 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG [...] 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "14298767360" 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:20 INFO [http_cmd]: http status code OK 08:29:20 DEBUG --- status code --- 08:29:20 DEBUG http status code is 200 (expected 200) 08:29:20 DEBUG --- http response --- 08:29:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:29:20 DEBUG ----- output ----- 08:29:20 DEBUG { 08:29:20 DEBUG "status": "success", 08:29:20 DEBUG "data": { 08:29:20 DEBUG "resultType": "vector", 08:29:20 DEBUG "result": [ 08:29:20 DEBUG { 08:29:20 DEBUG "metric": { 08:29:20 DEBUG "pod": "admin-ui-7657fd47d8-xl9bm" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG [...] 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG 08:29:20 DEBUG "pod": "overseer-0-5b849496f7-qmtsz" 08:29:20 DEBUG }, 08:29:20 DEBUG "value": [ 08:29:20 DEBUG 1686554960, 08:29:20 DEBUG "237509.91672025176" 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG ] 08:29:20 DEBUG } 08:29:20 DEBUG } 08:29:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686554961" 08:29:21 INFO [http_cmd]: http status code OK 08:29:21 DEBUG --- status code --- 08:29:21 DEBUG http status code is 200 (expected 200) 08:29:21 DEBUG --- http response --- 08:29:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:21 DEBUG ----- output ----- 08:29:21 DEBUG { 08:29:21 DEBUG "status": "success", 08:29:21 DEBUG "data": { 08:29:21 DEBUG "resultType": "vector", 08:29:21 DEBUG "result": [ 08:29:21 DEBUG { 08:29:21 DEBUG "metric": { 08:29:21 DEBUG "instance": "10.142.0.100:9100" 08:29:21 DEBUG }, 08:29:21 DEBUG "value": [ 08:29:21 DEBUG 08:29:21 DEBUG 08:29:21 DEBUG 08:29:21 DEBUG [...] 08:29:21 DEBUG 08:29:21 DEBUG 08:29:21 DEBUG 08:29:21 DEBUG "instance": "10.142.0.99:9100" 08:29:21 DEBUG }, 08:29:21 DEBUG "value": [ 08:29:21 DEBUG 1686554961, 08:29:21 DEBUG "4460.133333333333" 08:29:21 DEBUG ] 08:29:21 DEBUG } 08:29:21 DEBUG ] 08:29:21 DEBUG } 08:29:21 DEBUG } 08:29:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686554961" 08:29:21 INFO [http_cmd]: http status code OK 08:29:21 DEBUG --- status code --- 08:29:21 DEBUG http status code is 200 (expected 200) 08:29:21 DEBUG --- http response --- 08:29:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:29:21 DEBUG ----- output ----- 08:29:21 DEBUG { 08:29:21 DEBUG "status": "success", 08:29:21 DEBUG "data": { 08:29:21 DEBUG "resultType": "vector", 08:29:21 DEBUG "result": [ 08:29:21 DEBUG { 08:29:21 DEBUG "metric": { 08:29:21 DEBUG "pod": "ds-cts-2" 08:29:21 DEBUG }, 08:29:21 DEBUG "value": [ 08:29:21 DEBUG 08:29:21 DEBUG 08:29:21 DEBUG 08:29:21 DEBUG [...] 08:29:21 DEBUG 08:29:21 DEBUG 08:29:21 DEBUG 08:29:21 DEBUG "pod": "ds-idrepo-1" 08:29:21 DEBUG }, 08:29:21 DEBUG "value": [ 08:29:21 DEBUG 1686554961, 08:29:21 DEBUG "0" 08:29:21 DEBUG ] 08:29:21 DEBUG } 08:29:21 DEBUG ] 08:29:21 DEBUG } 08:29:21 DEBUG } 127.0.0.1 - - [12/Jun/2023 08:29:25] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:26] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:27] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:28] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:35] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:36] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:36] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:37] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:44] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:45] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:46] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:47] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:53] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:54] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 08:29:54 INFO 08:29:54 INFO [loop_until]: kubectl --namespace=xlou top pods 08:29:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:29:54 INFO [loop_until]: OK (rc = 0) 08:29:54 DEBUG --- stdout --- 08:29:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-xl9bm 1m 5Mi am-7849cf7bdb-bd9j2 9m 6125Mi am-7849cf7bdb-npvzp 6m 6234Mi am-7849cf7bdb-p4fns 8m 6286Mi ds-cts-0 10m 416Mi ds-cts-1 8m 451Mi ds-cts-2 10m 371Mi ds-idrepo-0 16m 13585Mi ds-idrepo-1 17m 13545Mi ds-idrepo-2 17m 13560Mi end-user-ui-5c589c665b-tgm8n 1m 5Mi idm-58dc667486-ml9mc 4m 5458Mi idm-58dc667486-vrqsf 5m 5518Mi lodemon-75f677b97d-6b58v 91m 80Mi login-ui-55f8bc5458-nxdzc 1m 3Mi overseer-0-5b849496f7-qmtsz 1000m 3785Mi 08:29:54 DEBUG --- stderr --- 08:29:54 DEBUG 127.0.0.1 - - [12/Jun/2023 08:29:55] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:29:56] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:30:03] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:30:04] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:30:05] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:30:06] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 08:30:07 INFO 08:30:07 INFO [loop_until]: kubectl --namespace=xlou top node 08:30:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:30:07 INFO [loop_until]: OK (rc = 0) 08:30:07 DEBUG --- stdout --- 08:30:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 65m 0% 6414Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 7166Mi 12% gke-xlou-cdm-default-pool-f05840a3-83ll 65m 0% 7056Mi 12% gke-xlou-cdm-default-pool-f05840a3-8bck 67m 0% 6414Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 66m 0% 7378Mi 12% gke-xlou-cdm-default-pool-f05840a3-m0lk 234m 1% 3029Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 195m 1% 1218Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 69m 0% 1121Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1155Mi 1% gke-xlou-cdm-ds-32e4dcb1-5msf 75m 0% 14531Mi 24% gke-xlou-cdm-ds-32e4dcb1-jr09 76m 0% 14528Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 68m 0% 1052Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 67m 0% 14586Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1076m 6% 4643Mi 7% 08:30:07 DEBUG --- stderr --- 08:30:07 DEBUG 127.0.0.1 - - [12/Jun/2023 08:30:12] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:30:13] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:30:14] "GET /monitoring/average?start_time=23-06-12_02:42:03&stop_time=23-06-12_04:10:38 HTTP/1.1" 200 - 08:30:14 INFO Finished: True 08:30:14 INFO Waiting for threads to register finish flag 08:31:07 INFO Done. Have a nice day! :) 127.0.0.1 - - [12/Jun/2023 08:31:07] "GET /monitoring/stop HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:31:07] "GET /monitoring/average?start_time=23-06-12_04:14:38&stop_time=23-06-12_05:43:19 HTTP/1.1" 200 - 08:31:15 INFO File /tmp/lodemon_data-23-06-11_19:41:54/am_cts_reaper_cache_size.json does not exist. Skipping... 08:31:15 INFO File /tmp/lodemon_data-23-06-11_19:41:54/am_cts_reaper_search_seconds_total.json does not exist. Skipping... 08:31:15 INFO File /tmp/lodemon_data-23-06-11_19:41:54/ds_replication_replica_replayed_updates_conflicts_resolved.json does not exist. Skipping... 08:31:15 INFO File /tmp/lodemon_data-23-06-11_19:41:54/node_disk_read_bytes_total.json does not exist. Skipping... 08:31:15 INFO File /tmp/lodemon_data-23-06-11_19:41:54/node_disk_written_bytes_total.json does not exist. Skipping... 08:31:15 INFO File /tmp/lodemon_data-23-06-11_19:41:54/ds_backend_entry_count.json does not exist. Skipping... 127.0.0.1 - - [12/Jun/2023 08:31:24] "GET /monitoring/process HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:31:24] "GET /monitoring/average?start_time=23-06-11_19:07:57&stop_time=23-06-12_01:05:22 HTTP/1.1" 200 - 127.0.0.1 - - [12/Jun/2023 08:31:25] "GET /monitoring/average?start_time=23-06-12_01:09:22&stop_time=23-06-12_02:38:03 HTTP/1.1" 200 -