==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-6455d5ff7c-zzndq Namespace: xlou Priority: 0 Node: gke-xlou-cdm-default-pool-f05840a3-s6rh/10.142.0.104 Start Time: Tue, 13 Jun 2023 15:31:52 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=6455d5ff7c skaffold.dev/run-id=61c6006f-2295-4e3b-8e9b-5232ff175ccb Annotations: Status: Running IP: 10.106.46.66 IPs: IP: 10.106.46.66 Controlled By: ReplicaSet/lodemon-6455d5ff7c Containers: lodemon: Container ID: containerd://f89040a033c11d2f0f937447fb2b1a6aabd9a416a0a5840e22d9ee20d6f8b045 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:825a0581d069cb8056effcc909c4108144200594e5f34977359ab4df875932ac Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Tue, 13 Jun 2023 15:32:09 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-q7lql (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-q7lql: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 21:27:14 INFO 21:27:14 INFO [loop_until]: kubectl --namespace=xlou top pods 21:27:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:27:14 INFO [loop_until]: OK (rc = 0) 21:27:14 DEBUG --- stdout --- 21:27:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5754Mi am-7849cf7bdb-ntqzs 11m 5696Mi am-7849cf7bdb-xrngh 9m 5683Mi ds-cts-0 15m 416Mi ds-cts-1 9m 421Mi ds-cts-2 9m 392Mi ds-idrepo-0 3402m 13797Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3127m 4810Mi idm-58dc667486-rgz9j 2939m 5087Mi lodemon-6455d5ff7c-zzndq 4m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 213m 636Mi 21:27:14 DEBUG --- stderr --- 21:27:14 DEBUG 21:27:17 INFO 21:27:17 INFO [loop_until]: kubectl --namespace=xlou top node 21:27:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:27:17 INFO [loop_until]: OK (rc = 0) 21:27:17 DEBUG --- stdout --- 21:27:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3146m 19% 5761Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6810Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2856m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6943Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 436m 2% 3139Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 88m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13701Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3602m 22% 14513Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 68m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14580Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 285m 1% 1950Mi 3% 21:27:17 DEBUG --- stderr --- 21:27:17 DEBUG 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG { 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG "data": { 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG "result": [ 21:27:59 DEBUG { 21:27:59 DEBUG "metric": { 21:27:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG [...] 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG "0" 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG } 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG { 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG "data": { 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG "result": [ 21:27:59 DEBUG { 21:27:59 DEBUG "metric": { 21:27:59 DEBUG "pod": "ds-cts-0" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG [...] 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "pod": "ds-idrepo-0" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG "587.3" 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG } 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688079" 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG { 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG "data": { 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG "result": [ 21:27:59 DEBUG { 21:27:59 DEBUG "metric": { 21:27:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG [...] 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG "0" 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG } 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG { 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG "data": { 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG "result": [ 21:27:59 DEBUG { 21:27:59 DEBUG "metric": { 21:27:59 DEBUG "instance": "10.142.0.100:9100" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG [...] 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "instance": "10.142.0.99:9100" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG "0" 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG } 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG { 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG "data": { 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG "result": [ 21:27:59 DEBUG { 21:27:59 DEBUG "metric": { 21:27:59 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG [...] 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG "0" 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG } 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG { 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG "data": { 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG "result": [ 21:27:59 DEBUG { 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG "metric": { 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG }, 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG 21:27:59 DEBUG { 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG [...] 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG 21:27:59 DEBUG "data": { 21:27:59 DEBUG 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG "result": [ 21:27:59 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:27:59 DEBUG { 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG { 21:27:59 DEBUG }, 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG "metric": { 21:27:59 DEBUG "value": [ 21:27:59 DEBUG "data": { 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:27:59 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG }, 21:27:59 DEBUG "0" 21:27:59 DEBUG "result": [ 21:27:59 DEBUG { 21:27:59 DEBUG "value": [ 21:27:59 DEBUG ] 21:27:59 DEBUG { 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG 21:27:59 DEBUG } 21:27:59 DEBUG "metric": { 21:27:59 DEBUG "data": { 21:27:59 DEBUG 21:27:59 DEBUG ] 21:27:59 DEBUG "pod": "ds-cts-0" 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG 21:27:59 DEBUG } 21:27:59 DEBUG }, 21:27:59 DEBUG "result": [ 21:27:59 DEBUG [...] 21:27:59 DEBUG } 21:27:59 DEBUG "value": [ 21:27:59 DEBUG { 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "metric": { 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "pod": "ds-idrepo-2" 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG }, 21:27:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:27:59 DEBUG [...] 21:27:59 DEBUG "value": [ 21:27:59 DEBUG }, 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "0" 21:27:59 DEBUG "pod": "ds-cts-2" 21:27:59 DEBUG [...] 21:27:59 DEBUG ] 21:27:59 DEBUG }, 21:27:59 DEBUG 21:27:59 DEBUG } 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 21:27:59 DEBUG ] 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG 21:27:59 DEBUG } 21:27:59 DEBUG "0" 21:27:59 DEBUG "pod": "ds-idrepo-0" 21:27:59 DEBUG } 21:27:59 DEBUG ] 21:27:59 DEBUG }, 21:27:59 DEBUG } 21:27:59 DEBUG "value": [ 21:27:59 DEBUG ] 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG } 21:27:59 DEBUG "587.2666666666667" 21:27:59 DEBUG } 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG } 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688079" 21:27:59 INFO [http_cmd]: http status code OK 21:27:59 DEBUG --- status code --- 21:27:59 DEBUG http status code is 200 (expected 200) 21:27:59 DEBUG --- http response --- 21:27:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:27:59 DEBUG ----- output ----- 21:27:59 DEBUG { 21:27:59 DEBUG "status": "success", 21:27:59 DEBUG "data": { 21:27:59 DEBUG "resultType": "vector", 21:27:59 DEBUG "result": [ 21:27:59 DEBUG { 21:27:59 DEBUG "metric": { 21:27:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG [...] 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG 21:27:59 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:27:59 DEBUG }, 21:27:59 DEBUG "value": [ 21:27:59 DEBUG 1686688079, 21:27:59 DEBUG "0.2" 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG ] 21:27:59 DEBUG } 21:27:59 DEBUG } 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG { 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "data": { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG { 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "pod": "ds-idrepo-0" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 1686688079, 21:28:00 DEBUG "6620.4928170488065" 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688080" 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG { 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "data": { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG { 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "pod": "ds-cts-0" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "pod": "ds-idrepo-0" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG "0" 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688080" 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG { 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "data": { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG { 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG "13616996352" 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688080" 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG { 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "data": { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG { 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG "0" 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688080" 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG { 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "data": { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG { 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "pod": "ds-idrepo-2" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "pod": "ds-idrepo-0" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG "587.2666666666667" 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688080" 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG { 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "data": { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG { 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "instance": "10.142.0.100:9100" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "instance": "10.142.0.99:9100" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG "19704793.4" 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688080" 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG { 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "data": { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG { 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "instance": "10.142.0.100:9100" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "instance": "10.142.0.99:9100" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG "0.00040000000000001514" 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688080" 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG { 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "data": { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG { 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "instance": "10.142.0.100:9100" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "instance": "10.142.0.99:9100" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG "19706647.666666664" 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688080" 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG { 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "data": { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG { 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "pod": "ds-idrepo-2" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "pod": "ds-idrepo-0" 21:28:00 DEBUG }, 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG "587.2666666666667" 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688080" 21:28:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688080" 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 INFO [http_cmd]: http status code OK 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG --- status code --- 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:28:00 DEBUG http status code is 200 (expected 200) 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG --- http response --- 21:28:00 DEBUG { 21:28:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG ----- output ----- 21:28:00 DEBUG "data": { 21:28:00 DEBUG { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "status": "success", 21:28:00 DEBUG "result": [ 21:28:00 DEBUG "data": { 21:28:00 DEBUG { 21:28:00 DEBUG "resultType": "vector", 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "result": [ 21:28:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:28:00 DEBUG { 21:28:00 DEBUG }, 21:28:00 DEBUG "metric": { 21:28:00 DEBUG "value": [ 21:28:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:28:00 DEBUG 21:28:00 DEBUG }, 21:28:00 DEBUG 21:28:00 DEBUG "value": [ 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG [...] 21:28:00 DEBUG 21:28:00 DEBUG 21:28:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:28:00 DEBUG 21:28:00 DEBUG }, 21:28:00 DEBUG 21:28:00 DEBUG "value": [ 21:28:00 DEBUG "pod": "ds-idrepo-0" 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG }, 21:28:00 DEBUG "0.23354371676754013" 21:28:00 DEBUG "value": [ 21:28:00 DEBUG ] 21:28:00 DEBUG 1686688080, 21:28:00 DEBUG } 21:28:00 DEBUG "0.7611074112834174" 21:28:00 DEBUG ] 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:00 DEBUG ] 21:28:00 DEBUG } 21:28:00 DEBUG } 21:28:14 INFO 21:28:14 INFO [loop_until]: kubectl --namespace=xlou top pods 21:28:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:28:14 INFO [loop_until]: OK (rc = 0) 21:28:14 DEBUG --- stdout --- 21:28:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 9m 5696Mi am-7849cf7bdb-xrngh 10m 5683Mi ds-cts-0 13m 416Mi ds-cts-1 9m 422Mi ds-cts-2 13m 392Mi ds-idrepo-0 3053m 13800Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2729m 4811Mi idm-58dc667486-rgz9j 2535m 5088Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 192m 636Mi 21:28:14 DEBUG --- stderr --- 21:28:14 DEBUG 21:28:17 INFO 21:28:17 INFO [loop_until]: kubectl --namespace=xlou top node 21:28:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:28:17 INFO [loop_until]: OK (rc = 0) 21:28:17 DEBUG --- stdout --- 21:28:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2823m 17% 5763Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 72m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2517m 15% 6075Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 75m 0% 6944Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 433m 2% 3140Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 87m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 66m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1193Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13703Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3164m 19% 14511Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 69m 0% 14578Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 259m 1% 1951Mi 3% 21:28:17 DEBUG --- stderr --- 21:28:17 DEBUG 21:28:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688139" 21:28:59 INFO [http_cmd]: http status code OK 21:28:59 DEBUG --- status code --- 21:28:59 DEBUG http status code is 200 (expected 200) 21:28:59 DEBUG --- http response --- 21:28:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:28:59 DEBUG ----- output ----- 21:28:59 DEBUG { 21:28:59 DEBUG "status": "success", 21:28:59 DEBUG "data": { 21:28:59 DEBUG "resultType": "vector", 21:28:59 DEBUG "result": [ 21:28:59 DEBUG { 21:28:59 DEBUG "metric": { 21:28:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG [...] 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 1686688139, 21:28:59 DEBUG "0" 21:28:59 DEBUG ] 21:28:59 DEBUG } 21:28:59 DEBUG ] 21:28:59 DEBUG } 21:28:59 DEBUG } 21:28:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688139" 21:28:59 INFO [http_cmd]: http status code OK 21:28:59 DEBUG --- status code --- 21:28:59 DEBUG http status code is 200 (expected 200) 21:28:59 DEBUG --- http response --- 21:28:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:28:59 DEBUG ----- output ----- 21:28:59 DEBUG { 21:28:59 DEBUG "status": "success", 21:28:59 DEBUG "data": { 21:28:59 DEBUG "resultType": "vector", 21:28:59 DEBUG "result": [ 21:28:59 DEBUG { 21:28:59 DEBUG "metric": { 21:28:59 DEBUG "pod": "ds-cts-0" 21:28:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688139" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG [...] 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG "pod": "ds-idrepo-0" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 1686688139, 21:28:59 DEBUG "604" 21:28:59 DEBUG ] 21:28:59 DEBUG } 21:28:59 DEBUG ] 21:28:59 DEBUG } 21:28:59 DEBUG } 21:28:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688139" 21:28:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688139" 21:28:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688139" 21:28:59 INFO [http_cmd]: http status code OK 21:28:59 DEBUG --- status code --- 21:28:59 DEBUG http status code is 200 (expected 200) 21:28:59 DEBUG --- http response --- 21:28:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:28:59 DEBUG ----- output ----- 21:28:59 DEBUG { 21:28:59 DEBUG "status": "success", 21:28:59 DEBUG "data": { 21:28:59 DEBUG "resultType": "vector", 21:28:59 DEBUG "result": [ 21:28:59 DEBUG { 21:28:59 DEBUG "metric": { 21:28:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG [...] 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 1686688139, 21:28:59 DEBUG "0" 21:28:59 DEBUG ] 21:28:59 DEBUG } 21:28:59 DEBUG ] 21:28:59 DEBUG } 21:28:59 DEBUG } 21:28:59 INFO [http_cmd]: http status code OK 21:28:59 DEBUG --- status code --- 21:28:59 DEBUG http status code is 200 (expected 200) 21:28:59 DEBUG --- http response --- 21:28:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:28:59 DEBUG ----- output ----- 21:28:59 DEBUG { 21:28:59 DEBUG "status": "success", 21:28:59 DEBUG "data": { 21:28:59 DEBUG "resultType": "vector", 21:28:59 DEBUG "result": [ 21:28:59 DEBUG { 21:28:59 DEBUG "metric": { 21:28:59 DEBUG "instance": "10.142.0.100:9100" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG [...] 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG "instance": "10.142.0.99:9100" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 1686688139, 21:28:59 DEBUG "0.00033333333333333365" 21:28:59 DEBUG ] 21:28:59 DEBUG } 21:28:59 DEBUG ] 21:28:59 DEBUG } 21:28:59 DEBUG } 21:28:59 INFO [http_cmd]: http status code OK 21:28:59 DEBUG --- status code --- 21:28:59 DEBUG http status code is 200 (expected 200) 21:28:59 DEBUG --- http response --- 21:28:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:28:59 DEBUG ----- output ----- 21:28:59 DEBUG { 21:28:59 DEBUG "status": "success", 21:28:59 DEBUG "data": { 21:28:59 DEBUG "resultType": "vector", 21:28:59 DEBUG "result": [ 21:28:59 DEBUG { 21:28:59 DEBUG "metric": { 21:28:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG [...] 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG 21:28:59 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:28:59 DEBUG }, 21:28:59 DEBUG "value": [ 21:28:59 DEBUG 1686688139, 21:29:00 DEBUG "0" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "pod": "am-7849cf7bdb-lzlql" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688139, 21:29:00 DEBUG "0" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG { 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "data": { 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG "result": [ 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG { 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "pod": "ds-cts-0" 21:29:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:29:00 DEBUG "pod": "ds-idrepo-2" 21:29:00 DEBUG }, 21:29:00 DEBUG }, 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG "value": [ 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG [...] 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "pod": "ds-cts-2" 21:29:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:29:00 DEBUG "pod": "ds-idrepo-0" 21:29:00 DEBUG }, 21:29:00 DEBUG }, 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG "value": [ 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "0" 21:29:00 DEBUG "0.004562610048290924" 21:29:00 DEBUG "603.9666666666667" 21:29:00 DEBUG ] 21:29:00 DEBUG ] 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG ] 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "0.2" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "0" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "pod": "ds-cts-0" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "pod": "ds-idrepo-0" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "0" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688140" 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "13616033792" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "0" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "pod": "ds-idrepo-2" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "pod": "ds-idrepo-0" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "603.9666666666667" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688140" 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688140" 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "instance": "10.142.0.100:9100" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "instance": "10.142.0.99:9100" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "18656560.6" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "instance": "10.142.0.100:9100" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "instance": "10.142.0.99:9100" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "0.00040000000000001514" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688140" 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "instance": "10.142.0.100:9100" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "instance": "10.142.0.99:9100" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "18658805.233333334" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG "data": { 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG "result": [ 21:29:00 DEBUG { 21:29:00 DEBUG "metric": { 21:29:00 DEBUG "pod": "ds-idrepo-2" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG [...] 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "pod": "ds-idrepo-0" 21:29:00 DEBUG }, 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "603.9666666666667" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688140" 21:29:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688140" 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG { 21:29:00 INFO [http_cmd]: http status code OK 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG --- status code --- 21:29:00 DEBUG "data": { 21:29:00 DEBUG http status code is 200 (expected 200) 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG --- http response --- 21:29:00 DEBUG "result": [ 21:29:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:29:00 DEBUG { 21:29:00 DEBUG ----- output ----- 21:29:00 DEBUG "metric": { 21:29:00 DEBUG { 21:29:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:29:00 DEBUG "status": "success", 21:29:00 DEBUG }, 21:29:00 DEBUG "data": { 21:29:00 DEBUG "value": [ 21:29:00 DEBUG "resultType": "vector", 21:29:00 DEBUG 21:29:00 DEBUG "result": [ 21:29:00 DEBUG 21:29:00 DEBUG { 21:29:00 DEBUG 21:29:00 DEBUG "metric": { 21:29:00 DEBUG [...] 21:29:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:29:00 DEBUG 21:29:00 DEBUG }, 21:29:00 DEBUG 21:29:00 DEBUG "value": [ 21:29:00 DEBUG 21:29:00 DEBUG 21:29:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:29:00 DEBUG 21:29:00 DEBUG }, 21:29:00 DEBUG 21:29:00 DEBUG "value": [ 21:29:00 DEBUG [...] 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG 21:29:00 DEBUG "0" 21:29:00 DEBUG 21:29:00 DEBUG ] 21:29:00 DEBUG 21:29:00 DEBUG } 21:29:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:29:00 DEBUG ] 21:29:00 DEBUG }, 21:29:00 DEBUG } 21:29:00 DEBUG "value": [ 21:29:00 DEBUG } 21:29:00 DEBUG 1686688140, 21:29:00 DEBUG "0.2224131967122131" 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG ] 21:29:00 DEBUG } 21:29:00 DEBUG } 21:29:14 INFO 21:29:14 INFO [loop_until]: kubectl --namespace=xlou top pods 21:29:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:29:14 INFO [loop_until]: OK (rc = 0) 21:29:14 DEBUG --- stdout --- 21:29:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 9m 5696Mi am-7849cf7bdb-xrngh 10m 5683Mi ds-cts-0 9m 415Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3276m 13804Mi ds-idrepo-1 18m 13820Mi ds-idrepo-2 17m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3189m 4813Mi idm-58dc667486-rgz9j 2664m 5087Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 202m 636Mi 21:29:14 DEBUG --- stderr --- 21:29:14 DEBUG 21:29:17 INFO 21:29:17 INFO [loop_until]: kubectl --namespace=xlou top node 21:29:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:29:17 INFO [loop_until]: OK (rc = 0) 21:29:17 DEBUG --- stdout --- 21:29:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3191m 20% 5764Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6813Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2833m 17% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 75m 0% 6945Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 430m 2% 3132Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 87m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1194Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13705Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3299m 20% 14493Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 77m 0% 14577Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 263m 1% 1952Mi 3% 21:29:17 DEBUG --- stderr --- 21:29:17 DEBUG 21:29:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688199" 21:29:59 INFO [http_cmd]: http status code OK 21:29:59 DEBUG --- status code --- 21:29:59 DEBUG http status code is 200 (expected 200) 21:29:59 DEBUG --- http response --- 21:29:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:29:59 DEBUG ----- output ----- 21:29:59 DEBUG { 21:29:59 DEBUG "status": "success", 21:29:59 DEBUG "data": { 21:29:59 DEBUG "resultType": "vector", 21:29:59 DEBUG "result": [ 21:29:59 DEBUG { 21:29:59 DEBUG "metric": { 21:29:59 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:29:59 DEBUG }, 21:29:59 DEBUG "value": [ 21:29:59 DEBUG 21:29:59 DEBUG 21:29:59 DEBUG 21:29:59 DEBUG [...] 21:29:59 DEBUG 21:29:59 DEBUG 21:29:59 DEBUG 21:29:59 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:29:59 DEBUG }, 21:29:59 DEBUG "value": [ 21:29:59 DEBUG 1686688199, 21:29:59 DEBUG "0" 21:29:59 DEBUG ] 21:29:59 DEBUG } 21:29:59 DEBUG ] 21:29:59 DEBUG } 21:29:59 DEBUG } 21:29:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688199" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "pod": "ds-cts-0" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "pod": "ds-idrepo-0" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688199, 21:30:00 DEBUG "590.2" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688200" 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "data": { 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:30:00 DEBUG }, 21:30:00 DEBUG "result": [ 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG { 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG "metric": { 21:30:00 DEBUG { 21:30:00 DEBUG 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG 21:30:00 DEBUG "instance": "10.142.0.100:9100" 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG 21:30:00 DEBUG }, 21:30:00 DEBUG "data": { 21:30:00 DEBUG 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG [...] 21:30:00 DEBUG "value": [ 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG [...] 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "result": [ 21:30:00 DEBUG 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG { 21:30:00 DEBUG 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "metric": { 21:30:00 DEBUG 21:30:00 DEBUG { 21:30:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:30:00 DEBUG [...] 21:30:00 DEBUG "pod": "ds-cts-2" 21:30:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG }, 21:30:00 DEBUG 21:30:00 DEBUG }, 21:30:00 DEBUG }, 21:30:00 DEBUG "data": { 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG "value": [ 21:30:00 DEBUG "value": [ 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "result": [ 21:30:00 DEBUG "0" 21:30:00 DEBUG "instance": "10.142.0.99:9100" 21:30:00 DEBUG 21:30:00 DEBUG "0" 21:30:00 DEBUG { 21:30:00 DEBUG ] 21:30:00 DEBUG }, 21:30:00 DEBUG 21:30:00 DEBUG ] 21:30:00 DEBUG "metric": { 21:30:00 DEBUG } 21:30:00 DEBUG "value": [ 21:30:00 DEBUG [...] 21:30:00 DEBUG } 21:30:00 DEBUG "pod": "ds-cts-2" 21:30:00 DEBUG ] 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG 21:30:00 DEBUG ] 21:30:00 DEBUG }, 21:30:00 DEBUG } 21:30:00 DEBUG "0.0003333333333333558" 21:30:00 DEBUG 21:30:00 DEBUG } 21:30:00 DEBUG "value": [ 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG 21:30:00 DEBUG } 21:30:00 DEBUG 21:30:00 DEBUG } 21:30:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:30:00 DEBUG 21:30:00 DEBUG ] 21:30:00 DEBUG }, 21:30:00 DEBUG 21:30:00 DEBUG } 21:30:00 DEBUG "value": [ 21:30:00 DEBUG [...] 21:30:00 DEBUG } 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG 21:30:00 DEBUG "0" 21:30:00 DEBUG 21:30:00 DEBUG ] 21:30:00 DEBUG 21:30:00 DEBUG } 21:30:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:30:00 DEBUG ] 21:30:00 DEBUG }, 21:30:00 DEBUG } 21:30:00 DEBUG "value": [ 21:30:00 DEBUG } 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "0.00003103766260300958" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "pod": "ds-cts-2" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "0" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG { 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG "data": { 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "data": { 21:30:00 DEBUG "pod": "ds-cts-0" 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG }, 21:30:00 DEBUG "result": [ 21:30:00 DEBUG "value": [ 21:30:00 DEBUG { 21:30:00 DEBUG 21:30:00 DEBUG "metric": { 21:30:00 DEBUG 21:30:00 DEBUG "pod": "ds-idrepo-2" 21:30:00 DEBUG 21:30:00 DEBUG }, 21:30:00 DEBUG [...] 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "pod": "ds-cts-2" 21:30:00 DEBUG [...] 21:30:00 DEBUG }, 21:30:00 DEBUG 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG 21:30:00 DEBUG "0" 21:30:00 DEBUG "pod": "ds-idrepo-0" 21:30:00 DEBUG ] 21:30:00 DEBUG }, 21:30:00 DEBUG } 21:30:00 DEBUG "value": [ 21:30:00 DEBUG ] 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG } 21:30:00 DEBUG "590.1666666666666" 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "0.2" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "pod": "ds-cts-0" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "pod": "ds-idrepo-0" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "0" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "13616988160" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "0" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "pod": "ds-idrepo-2" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "pod": "ds-idrepo-0" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "590.1666666666666" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "instance": "10.142.0.100:9100" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "instance": "10.142.0.99:9100" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "19702450.133333333" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "instance": "10.142.0.100:9100" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "instance": "10.142.0.99:9100" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "0.0004999999999995453" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "instance": "10.142.0.100:9100" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "instance": "10.142.0.99:9100" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "19704347.1" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "pod": "ds-idrepo-2" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "pod": "ds-idrepo-0" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "590.1333333333333" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "0.22168365072143126" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688200" 21:30:00 INFO [http_cmd]: http status code OK 21:30:00 DEBUG --- status code --- 21:30:00 DEBUG http status code is 200 (expected 200) 21:30:00 DEBUG --- http response --- 21:30:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:30:00 DEBUG ----- output ----- 21:30:00 DEBUG { 21:30:00 DEBUG "status": "success", 21:30:00 DEBUG "data": { 21:30:00 DEBUG "resultType": "vector", 21:30:00 DEBUG "result": [ 21:30:00 DEBUG { 21:30:00 DEBUG "metric": { 21:30:00 DEBUG "pod": "ds-cts-2" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG [...] 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG 21:30:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:30:00 DEBUG }, 21:30:00 DEBUG "value": [ 21:30:00 DEBUG 1686688200, 21:30:00 DEBUG "0" 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG ] 21:30:00 DEBUG } 21:30:00 DEBUG } 21:30:14 INFO 21:30:14 INFO [loop_until]: kubectl --namespace=xlou top pods 21:30:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:30:14 INFO [loop_until]: OK (rc = 0) 21:30:14 DEBUG --- stdout --- 21:30:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 9m 5696Mi am-7849cf7bdb-xrngh 9m 5683Mi ds-cts-0 9m 416Mi ds-cts-1 20m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3333m 13827Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2937m 4816Mi idm-58dc667486-rgz9j 2789m 5087Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 204m 636Mi 21:30:14 DEBUG --- stderr --- 21:30:14 DEBUG 21:30:17 INFO 21:30:17 INFO [loop_until]: kubectl --namespace=xlou top node 21:30:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:30:17 INFO [loop_until]: OK (rc = 0) 21:30:17 DEBUG --- stdout --- 21:30:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3036m 19% 5767Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2794m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6943Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 443m 2% 3137Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 66m 0% 1195Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13706Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3363m 21% 14513Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14577Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 278m 1% 1950Mi 3% 21:30:17 DEBUG --- stderr --- 21:30:17 DEBUG 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "0" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "ds-cts-0" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "ds-idrepo-0" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "627.6333333333333" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688260" 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG "result": [ 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG { 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG "metric": { 21:31:00 DEBUG --- http response --- 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG }, 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG "value": [ 21:31:00 DEBUG { 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:31:00 DEBUG 21:31:00 DEBUG "data": { 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG { 21:31:00 DEBUG [...] 21:31:00 DEBUG "result": [ 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG 21:31:00 DEBUG { 21:31:00 DEBUG "data": { 21:31:00 DEBUG 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG 21:31:00 DEBUG "instance": "10.142.0.100:9100" 21:31:00 DEBUG "result": [ 21:31:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:31:00 DEBUG }, 21:31:00 DEBUG { 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG 21:31:00 DEBUG }, 21:31:00 DEBUG "228826469.3452988" 21:31:00 DEBUG 21:31:00 DEBUG "value": [ 21:31:00 DEBUG ] 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG } 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG ] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG } 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG } 21:31:00 DEBUG "instance": "10.142.0.99:9100" 21:31:00 DEBUG 21:31:00 DEBUG }, 21:31:00 DEBUG 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:31:00 DEBUG "0" 21:31:00 DEBUG }, 21:31:00 DEBUG ] 21:31:00 DEBUG "value": [ 21:31:00 DEBUG } 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG ] 21:31:00 DEBUG "0" 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "0" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG { 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG "data": { 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "data": { 21:31:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG }, 21:31:00 DEBUG "result": [ 21:31:00 DEBUG "value": [ 21:31:00 DEBUG { 21:31:00 DEBUG 21:31:00 DEBUG "metric": { 21:31:00 DEBUG 21:31:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:31:00 DEBUG 21:31:00 DEBUG }, 21:31:00 DEBUG [...] 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:31:00 DEBUG [...] 21:31:00 DEBUG }, 21:31:00 DEBUG 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG 21:31:00 DEBUG "0.2" 21:31:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:31:00 DEBUG ] 21:31:00 DEBUG }, 21:31:00 DEBUG } 21:31:00 DEBUG "value": [ 21:31:00 DEBUG ] 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG } 21:31:00 DEBUG "2.7945531564756605" 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "ds-cts-0" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "ds-cts-2" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "0" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "0" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "ds-idrepo-2" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "ds-idrepo-0" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "627.6333333333333" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "ds-cts-0" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "ds-idrepo-0" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "0" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "13617737728" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "ds-idrepo-2" 21:31:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:31:00 DEBUG }, 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "ds-idrepo-0" 21:31:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:31:00 DEBUG }, 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "627.6333333333333" 21:31:00 DEBUG "0" 21:31:00 DEBUG ] 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "instance": "10.142.0.100:9100" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "instance": "10.142.0.99:9100" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "0.0002333333333316053" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "instance": "10.142.0.100:9100" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "instance": "10.142.0.99:9100" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "19052880.3" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "instance": "10.142.0.100:9100" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "instance": "10.142.0.99:9100" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "19055103.7" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "ds-idrepo-2" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "ds-idrepo-0" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "627.6333333333333" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "0.23437825016723204" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688260" 21:31:00 INFO [http_cmd]: http status code OK 21:31:00 DEBUG --- status code --- 21:31:00 DEBUG http status code is 200 (expected 200) 21:31:00 DEBUG --- http response --- 21:31:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:31:00 DEBUG ----- output ----- 21:31:00 DEBUG { 21:31:00 DEBUG "status": "success", 21:31:00 DEBUG "data": { 21:31:00 DEBUG "resultType": "vector", 21:31:00 DEBUG "result": [ 21:31:00 DEBUG { 21:31:00 DEBUG "metric": { 21:31:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG [...] 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG 21:31:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:31:00 DEBUG }, 21:31:00 DEBUG "value": [ 21:31:00 DEBUG 1686688260, 21:31:00 DEBUG "0" 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG ] 21:31:00 DEBUG } 21:31:00 DEBUG } 21:31:14 INFO 21:31:14 INFO [loop_until]: kubectl --namespace=xlou top pods 21:31:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:31:14 INFO [loop_until]: OK (rc = 0) 21:31:14 DEBUG --- stdout --- 21:31:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 8m 5696Mi am-7849cf7bdb-xrngh 10m 5683Mi ds-cts-0 9m 416Mi ds-cts-1 8m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3281m 13800Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2890m 4818Mi idm-58dc667486-rgz9j 2709m 5088Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 201m 636Mi 21:31:14 DEBUG --- stderr --- 21:31:14 DEBUG 21:31:17 INFO 21:31:17 INFO [loop_until]: kubectl --namespace=xlou top node 21:31:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:31:18 INFO [loop_until]: OK (rc = 0) 21:31:18 DEBUG --- stdout --- 21:31:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3206m 20% 5769Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6812Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2622m 16% 6075Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6944Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 429m 2% 3147Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1475Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 57m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1196Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13707Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3495m 21% 14495Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14578Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 267m 1% 1951Mi 3% 21:31:18 DEBUG --- stderr --- 21:31:18 DEBUG 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "ds-cts-0" 21:32:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:32:00 DEBUG }, 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "ds-idrepo-0" 21:32:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:32:00 DEBUG }, 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "598.6666666666666" 21:32:00 DEBUG "0" 21:32:00 DEBUG ] 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "instance": "10.142.0.100:9100" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "instance": "10.142.0.99:9100" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "0.0003333333333333262" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:32:00 DEBUG }, 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG "value": [ 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG "0" 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG ] 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:32:00 DEBUG } 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG ] 21:32:00 DEBUG { 21:32:00 DEBUG } 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG } 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "am-7849cf7bdb-lzlql" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "0" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "am-7849cf7bdb-lzlql" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "0" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "0" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "ds-idrepo-2" 21:32:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:32:00 DEBUG }, 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "ds-idrepo-0" 21:32:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:32:00 DEBUG }, 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "598.7" 21:32:00 DEBUG "0.2" 21:32:00 DEBUG ] 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "ds-cts-0" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "ds-cts-2" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "0" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "0" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "ds-cts-0" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "ds-idrepo-0" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "0" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "13618458624" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "ds-idrepo-2" 21:32:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:32:00 DEBUG }, 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "ds-idrepo-0" 21:32:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:32:00 DEBUG }, 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "598.7333333333333" 21:32:00 DEBUG "0" 21:32:00 DEBUG ] 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "instance": "10.142.0.100:9100" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "instance": "10.142.0.99:9100" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "0.00040000000000001514" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "instance": "10.142.0.100:9100" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "instance": "10.142.0.99:9100" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "21758101.799999997" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "instance": "10.142.0.100:9100" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "instance": "10.142.0.99:9100" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "21760317.96666667" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688320" 21:32:00 INFO [http_cmd]: http status code OK 21:32:00 DEBUG --- status code --- 21:32:00 DEBUG http status code is 200 (expected 200) 21:32:00 DEBUG --- http response --- 21:32:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:32:00 DEBUG ----- output ----- 21:32:00 DEBUG { 21:32:00 DEBUG "status": "success", 21:32:00 DEBUG "data": { 21:32:00 DEBUG "resultType": "vector", 21:32:00 DEBUG "result": [ 21:32:00 DEBUG { 21:32:00 DEBUG "metric": { 21:32:00 DEBUG "pod": "ds-idrepo-2" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG [...] 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG 21:32:00 DEBUG "pod": "ds-idrepo-0" 21:32:00 DEBUG }, 21:32:00 DEBUG "value": [ 21:32:00 DEBUG 1686688320, 21:32:00 DEBUG "598.7333333333333" 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG ] 21:32:00 DEBUG } 21:32:00 DEBUG } 21:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688321" 21:32:01 INFO [http_cmd]: http status code OK 21:32:01 DEBUG --- status code --- 21:32:01 DEBUG http status code is 200 (expected 200) 21:32:01 DEBUG --- http response --- 21:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:32:01 DEBUG ----- output ----- 21:32:01 DEBUG { 21:32:01 DEBUG "status": "success", 21:32:01 DEBUG "data": { 21:32:01 DEBUG "resultType": "vector", 21:32:01 DEBUG "result": [ 21:32:01 DEBUG { 21:32:01 DEBUG "metric": { 21:32:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:32:01 DEBUG }, 21:32:01 DEBUG "value": [ 21:32:01 DEBUG 21:32:01 DEBUG 21:32:01 DEBUG 21:32:01 DEBUG [...] 21:32:01 DEBUG 21:32:01 DEBUG 21:32:01 DEBUG 21:32:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:32:01 DEBUG }, 21:32:01 DEBUG "value": [ 21:32:01 DEBUG 1686688321, 21:32:01 DEBUG "0.23982700230679527" 21:32:01 DEBUG ] 21:32:01 DEBUG } 21:32:01 DEBUG ] 21:32:01 DEBUG } 21:32:01 DEBUG } 21:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688321" 21:32:01 INFO [http_cmd]: http status code OK 21:32:01 DEBUG --- status code --- 21:32:01 DEBUG http status code is 200 (expected 200) 21:32:01 DEBUG --- http response --- 21:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:32:01 DEBUG ----- output ----- 21:32:01 DEBUG { 21:32:01 DEBUG "status": "success", 21:32:01 DEBUG "data": { 21:32:01 DEBUG "resultType": "vector", 21:32:01 DEBUG "result": [ 21:32:01 DEBUG { 21:32:01 DEBUG "metric": { 21:32:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:32:01 DEBUG }, 21:32:01 DEBUG "value": [ 21:32:01 DEBUG 21:32:01 DEBUG 21:32:01 DEBUG 21:32:01 DEBUG [...] 21:32:01 DEBUG 21:32:01 DEBUG 21:32:01 DEBUG 21:32:01 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 21:32:01 DEBUG }, 21:32:01 DEBUG "value": [ 21:32:01 DEBUG 1686688321, 21:32:01 DEBUG "0" 21:32:01 DEBUG ] 21:32:01 DEBUG } 21:32:01 DEBUG ] 21:32:01 DEBUG } 21:32:01 DEBUG } 21:32:14 INFO 21:32:14 INFO [loop_until]: kubectl --namespace=xlou top pods 21:32:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:32:14 INFO [loop_until]: OK (rc = 0) 21:32:14 DEBUG --- stdout --- 21:32:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5754Mi am-7849cf7bdb-ntqzs 10m 5696Mi am-7849cf7bdb-xrngh 9m 5683Mi ds-cts-0 11m 416Mi ds-cts-1 9m 423Mi ds-cts-2 8m 392Mi ds-idrepo-0 3004m 13802Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2874m 4820Mi idm-58dc667486-rgz9j 2323m 5088Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 189m 636Mi 21:32:14 DEBUG --- stderr --- 21:32:14 DEBUG 21:32:18 INFO 21:32:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:32:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:32:18 INFO [loop_until]: OK (rc = 0) 21:32:18 DEBUG --- stdout --- 21:32:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2887m 18% 5770Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6813Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2524m 15% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 75m 0% 6944Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 431m 2% 3144Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 86m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1196Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13703Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3032m 19% 14492Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14580Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 260m 1% 1949Mi 3% 21:32:18 DEBUG --- stderr --- 21:32:18 DEBUG 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688380" 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "instance": "10.142.0.100:9100" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "instance": "10.142.0.99:9100" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "0.0003333333333333558" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "0" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "pod": "ds-cts-0" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "pod": "ds-idrepo-0" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "652.3333333333334" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "0" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:33:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:33:00 DEBUG }, 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 21:33:00 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 21:33:00 DEBUG }, 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "28266.803547494794" 21:33:00 DEBUG "0.1963704710981742" 21:33:00 DEBUG ] 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG { 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG "data": { 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:33:00 DEBUG "result": [ 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:33:00 DEBUG "data": { 21:33:00 DEBUG }, 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "value": [ 21:33:00 DEBUG "result": [ 21:33:00 DEBUG 21:33:00 DEBUG { 21:33:00 DEBUG 21:33:00 DEBUG "metric": { 21:33:00 DEBUG 21:33:00 DEBUG "pod": "ds-cts-0" 21:33:00 DEBUG [...] 21:33:00 DEBUG }, 21:33:00 DEBUG 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:33:00 DEBUG 21:33:00 DEBUG }, 21:33:00 DEBUG [...] 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG 21:33:00 DEBUG "0" 21:33:00 DEBUG 21:33:00 DEBUG ] 21:33:00 DEBUG "pod": "ds-cts-2" 21:33:00 DEBUG } 21:33:00 DEBUG }, 21:33:00 DEBUG ] 21:33:00 DEBUG "value": [ 21:33:00 DEBUG } 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG } 21:33:00 DEBUG "0" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:33:00 DEBUG "pod": "ds-idrepo-2" 21:33:00 DEBUG }, 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:33:00 DEBUG "pod": "ds-idrepo-0" 21:33:00 DEBUG }, 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "0.2" 21:33:00 DEBUG "652.3333333333334" 21:33:00 DEBUG ] 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "0" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "pod": "ds-cts-0" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "pod": "ds-idrepo-0" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "0" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "13615755264" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "pod": "ds-idrepo-2" 21:33:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:33:00 DEBUG }, 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "pod": "ds-idrepo-0" 21:33:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:33:00 DEBUG }, 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "652.3333333333334" 21:33:00 DEBUG "0" 21:33:00 DEBUG ] 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "instance": "10.142.0.100:9100" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "instance": "10.142.0.99:9100" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "0.0005333333333320903" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "instance": "10.142.0.100:9100" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "instance": "10.142.0.99:9100" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "19462052.133333333" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688380" 21:33:00 INFO [http_cmd]: http status code OK 21:33:00 DEBUG --- status code --- 21:33:00 DEBUG http status code is 200 (expected 200) 21:33:00 DEBUG --- http response --- 21:33:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:33:00 DEBUG ----- output ----- 21:33:00 DEBUG { 21:33:00 DEBUG "status": "success", 21:33:00 DEBUG "data": { 21:33:00 DEBUG "resultType": "vector", 21:33:00 DEBUG "result": [ 21:33:00 DEBUG { 21:33:00 DEBUG "metric": { 21:33:00 DEBUG "instance": "10.142.0.100:9100" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG [...] 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG 21:33:00 DEBUG "instance": "10.142.0.99:9100" 21:33:00 DEBUG }, 21:33:00 DEBUG "value": [ 21:33:00 DEBUG 1686688380, 21:33:00 DEBUG "19463649.733333334" 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG ] 21:33:00 DEBUG } 21:33:00 DEBUG } 21:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688381" 21:33:01 INFO [http_cmd]: http status code OK 21:33:01 DEBUG --- status code --- 21:33:01 DEBUG http status code is 200 (expected 200) 21:33:01 DEBUG --- http response --- 21:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:33:01 DEBUG ----- output ----- 21:33:01 DEBUG { 21:33:01 DEBUG "status": "success", 21:33:01 DEBUG "data": { 21:33:01 DEBUG "resultType": "vector", 21:33:01 DEBUG "result": [ 21:33:01 DEBUG { 21:33:01 DEBUG "metric": { 21:33:01 DEBUG "pod": "ds-idrepo-2" 21:33:01 DEBUG }, 21:33:01 DEBUG "value": [ 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG [...] 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG "pod": "ds-idrepo-0" 21:33:01 DEBUG }, 21:33:01 DEBUG "value": [ 21:33:01 DEBUG 1686688381, 21:33:01 DEBUG "652.3" 21:33:01 DEBUG ] 21:33:01 DEBUG } 21:33:01 DEBUG ] 21:33:01 DEBUG } 21:33:01 DEBUG } 21:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688381" 21:33:01 INFO [http_cmd]: http status code OK 21:33:01 DEBUG --- status code --- 21:33:01 DEBUG http status code is 200 (expected 200) 21:33:01 DEBUG --- http response --- 21:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:33:01 DEBUG ----- output ----- 21:33:01 DEBUG { 21:33:01 DEBUG "status": "success", 21:33:01 DEBUG "data": { 21:33:01 DEBUG "resultType": "vector", 21:33:01 DEBUG "result": [ 21:33:01 DEBUG { 21:33:01 DEBUG "metric": { 21:33:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:33:01 DEBUG }, 21:33:01 DEBUG "value": [ 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG [...] 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:33:01 DEBUG }, 21:33:01 DEBUG "value": [ 21:33:01 DEBUG 1686688381, 21:33:01 DEBUG "0.21915833838841442" 21:33:01 DEBUG ] 21:33:01 DEBUG } 21:33:01 DEBUG ] 21:33:01 DEBUG } 21:33:01 DEBUG } 21:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688381" 21:33:01 INFO [http_cmd]: http status code OK 21:33:01 DEBUG --- status code --- 21:33:01 DEBUG http status code is 200 (expected 200) 21:33:01 DEBUG --- http response --- 21:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:33:01 DEBUG ----- output ----- 21:33:01 DEBUG { 21:33:01 DEBUG "status": "success", 21:33:01 DEBUG "data": { 21:33:01 DEBUG "resultType": "vector", 21:33:01 DEBUG "result": [ 21:33:01 DEBUG { 21:33:01 DEBUG "metric": { 21:33:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:33:01 DEBUG }, 21:33:01 DEBUG "value": [ 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG [...] 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG 21:33:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:33:01 DEBUG }, 21:33:01 DEBUG "value": [ 21:33:01 DEBUG 1686688381, 21:33:01 DEBUG "0" 21:33:01 DEBUG ] 21:33:01 DEBUG } 21:33:01 DEBUG ] 21:33:01 DEBUG } 21:33:01 DEBUG } 21:33:14 INFO 21:33:14 INFO [loop_until]: kubectl --namespace=xlou top pods 21:33:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:33:14 INFO [loop_until]: OK (rc = 0) 21:33:14 DEBUG --- stdout --- 21:33:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5754Mi am-7849cf7bdb-ntqzs 10m 5696Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 423Mi ds-cts-2 9m 392Mi ds-idrepo-0 3223m 13802Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3023m 4822Mi idm-58dc667486-rgz9j 2434m 5088Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 204m 637Mi 21:33:14 DEBUG --- stderr --- 21:33:14 DEBUG 21:33:18 INFO 21:33:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:33:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:33:18 INFO [loop_until]: OK (rc = 0) 21:33:18 DEBUG --- stdout --- 21:33:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3201m 20% 5774Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6813Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2526m 15% 6073Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 75m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 450m 2% 3155Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13701Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3264m 20% 14492Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14580Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 253m 1% 1953Mi 3% 21:33:18 DEBUG --- stderr --- 21:33:18 DEBUG 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "0" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688440" 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "0" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "instance": "10.142.0.100:9100" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "instance": "10.142.0.99:9100" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "0" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "ds-cts-0" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "ds-idrepo-0" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "634.1" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "3472.6059713701434" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "0.0027529208182655047" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:34:00 DEBUG "pod": "ds-cts-0" 21:34:00 DEBUG }, 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:34:00 DEBUG "pod": "ds-cts-2" 21:34:00 DEBUG }, 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "0" 21:34:00 DEBUG "0" 21:34:00 DEBUG ] 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "ds-idrepo-2" 21:34:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:34:00 DEBUG }, 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "ds-idrepo-0" 21:34:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:34:00 DEBUG }, 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "634.1" 21:34:00 DEBUG "0.2" 21:34:00 DEBUG ] 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "0" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "ds-cts-0" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "ds-idrepo-0" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "0" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "13614551040" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "0" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "pod": "ds-idrepo-2" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "pod": "ds-idrepo-0" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "634.1" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "instance": "10.142.0.100:9100" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "instance": "10.142.0.99:9100" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "0.00036666666666557526" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "instance": "10.142.0.100:9100" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "instance": "10.142.0.99:9100" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "20447118.166666668" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688440" 21:34:00 INFO [http_cmd]: http status code OK 21:34:00 DEBUG --- status code --- 21:34:00 DEBUG http status code is 200 (expected 200) 21:34:00 DEBUG --- http response --- 21:34:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:34:00 DEBUG ----- output ----- 21:34:00 DEBUG { 21:34:00 DEBUG "status": "success", 21:34:00 DEBUG "data": { 21:34:00 DEBUG "resultType": "vector", 21:34:00 DEBUG "result": [ 21:34:00 DEBUG { 21:34:00 DEBUG "metric": { 21:34:00 DEBUG "instance": "10.142.0.100:9100" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG [...] 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG 21:34:00 DEBUG "instance": "10.142.0.99:9100" 21:34:00 DEBUG }, 21:34:00 DEBUG "value": [ 21:34:00 DEBUG 1686688440, 21:34:00 DEBUG "20449159.566666666" 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG ] 21:34:00 DEBUG } 21:34:00 DEBUG } 21:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688441" 21:34:01 INFO [http_cmd]: http status code OK 21:34:01 DEBUG --- status code --- 21:34:01 DEBUG http status code is 200 (expected 200) 21:34:01 DEBUG --- http response --- 21:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:34:01 DEBUG ----- output ----- 21:34:01 DEBUG { 21:34:01 DEBUG "status": "success", 21:34:01 DEBUG "data": { 21:34:01 DEBUG "resultType": "vector", 21:34:01 DEBUG "result": [ 21:34:01 DEBUG { 21:34:01 DEBUG "metric": { 21:34:01 DEBUG "pod": "ds-idrepo-2" 21:34:01 DEBUG }, 21:34:01 DEBUG "value": [ 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG [...] 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG "pod": "ds-idrepo-0" 21:34:01 DEBUG }, 21:34:01 DEBUG "value": [ 21:34:01 DEBUG 1686688441, 21:34:01 DEBUG "634.1" 21:34:01 DEBUG ] 21:34:01 DEBUG } 21:34:01 DEBUG ] 21:34:01 DEBUG } 21:34:01 DEBUG } 21:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688441" 21:34:01 INFO [http_cmd]: http status code OK 21:34:01 DEBUG --- status code --- 21:34:01 DEBUG http status code is 200 (expected 200) 21:34:01 DEBUG --- http response --- 21:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:34:01 DEBUG ----- output ----- 21:34:01 DEBUG { 21:34:01 DEBUG "status": "success", 21:34:01 DEBUG "data": { 21:34:01 DEBUG "resultType": "vector", 21:34:01 DEBUG "result": [ 21:34:01 DEBUG { 21:34:01 DEBUG "metric": { 21:34:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:34:01 DEBUG }, 21:34:01 DEBUG "value": [ 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG [...] 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:34:01 DEBUG }, 21:34:01 DEBUG "value": [ 21:34:01 DEBUG 1686688441, 21:34:01 DEBUG "0.22873474129185595" 21:34:01 DEBUG ] 21:34:01 DEBUG } 21:34:01 DEBUG ] 21:34:01 DEBUG } 21:34:01 DEBUG } 21:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688441" 21:34:01 INFO [http_cmd]: http status code OK 21:34:01 DEBUG --- status code --- 21:34:01 DEBUG http status code is 200 (expected 200) 21:34:01 DEBUG --- http response --- 21:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:34:01 DEBUG ----- output ----- 21:34:01 DEBUG { 21:34:01 DEBUG "status": "success", 21:34:01 DEBUG "data": { 21:34:01 DEBUG "resultType": "vector", 21:34:01 DEBUG "result": [ 21:34:01 DEBUG { 21:34:01 DEBUG "metric": { 21:34:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:34:01 DEBUG }, 21:34:01 DEBUG "value": [ 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG [...] 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG 21:34:01 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 21:34:01 DEBUG }, 21:34:01 DEBUG "value": [ 21:34:01 DEBUG 1686688441, 21:34:01 DEBUG "0" 21:34:01 DEBUG ] 21:34:01 DEBUG } 21:34:01 DEBUG ] 21:34:01 DEBUG } 21:34:01 DEBUG } 21:34:15 INFO 21:34:15 INFO [loop_until]: kubectl --namespace=xlou top pods 21:34:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:34:15 INFO [loop_until]: OK (rc = 0) 21:34:15 DEBUG --- stdout --- 21:34:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5754Mi am-7849cf7bdb-ntqzs 10m 5696Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 13m 416Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3535m 13803Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3269m 4823Mi idm-58dc667486-rgz9j 2933m 5088Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 210m 637Mi 21:34:15 DEBUG --- stderr --- 21:34:15 DEBUG 21:34:18 INFO 21:34:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:34:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:34:18 INFO [loop_until]: OK (rc = 0) 21:34:18 DEBUG --- stdout --- 21:34:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3378m 21% 5774Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6810Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2985m 18% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 449m 2% 3155Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1195Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13702Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3624m 22% 14514Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14582Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 272m 1% 1954Mi 3% 21:34:18 DEBUG --- stderr --- 21:34:18 DEBUG 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688500" 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:35:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:35:00 DEBUG }, 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG 21:35:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:35:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:35:00 DEBUG }, 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG { 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "0" 21:35:00 DEBUG "data": { 21:35:00 DEBUG "0" 21:35:00 DEBUG ] 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG "result": [ 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG { 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG "metric": { 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 DEBUG "instance": "10.142.0.100:9100" 21:35:00 DEBUG } 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG [...] 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:35:00 DEBUG 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG "instance": "10.142.0.99:9100" 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG { 21:35:00 DEBUG }, 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "value": [ 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:35:00 DEBUG "data": { 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "0" 21:35:00 DEBUG { 21:35:00 DEBUG "result": [ 21:35:00 DEBUG ] 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG { 21:35:00 DEBUG } 21:35:00 DEBUG "data": { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG ] 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "pod": "ds-cts-2" 21:35:00 DEBUG } 21:35:00 DEBUG "result": [ 21:35:00 DEBUG }, 21:35:00 DEBUG } 21:35:00 DEBUG { 21:35:00 DEBUG "value": [ 21:35:00 DEBUG "metric": { 21:35:00 DEBUG 21:35:00 DEBUG "pod": "ds-cts-0" 21:35:00 DEBUG 21:35:00 DEBUG }, 21:35:00 DEBUG 21:35:00 DEBUG "value": [ 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:35:00 DEBUG 21:35:00 DEBUG }, 21:35:00 DEBUG 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "pod": "ds-idrepo-0" 21:35:00 DEBUG "0" 21:35:00 DEBUG }, 21:35:00 DEBUG ] 21:35:00 DEBUG "value": [ 21:35:00 DEBUG } 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG ] 21:35:00 DEBUG "657.3333333333334" 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "pod": "ds-cts-2" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "0" 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "pod": "ds-cts-0" 21:35:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:35:00 DEBUG }, 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "pod": "ds-cts-2" 21:35:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:35:00 DEBUG }, 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "0" 21:35:00 DEBUG "0" 21:35:00 DEBUG ] 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "0.2" 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "pod": "ds-idrepo-2" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "pod": "ds-idrepo-0" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "657.4" 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "pod": "ds-cts-2" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "0" 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "pod": "ds-cts-0" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "pod": "ds-idrepo-0" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "0" 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "13615276032" 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "pod": "ds-idrepo-2" 21:35:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:35:00 DEBUG }, 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "pod": "ds-idrepo-0" 21:35:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:35:00 DEBUG }, 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "657.3666666666667" 21:35:00 DEBUG "0" 21:35:00 DEBUG ] 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "instance": "10.142.0.100:9100" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "instance": "10.142.0.99:9100" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "0.0004333333333325603" 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "instance": "10.142.0.100:9100" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "instance": "10.142.0.99:9100" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "20316598.666666664" 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688500" 21:35:00 INFO [http_cmd]: http status code OK 21:35:00 DEBUG --- status code --- 21:35:00 DEBUG http status code is 200 (expected 200) 21:35:00 DEBUG --- http response --- 21:35:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:35:00 DEBUG ----- output ----- 21:35:00 DEBUG { 21:35:00 DEBUG "status": "success", 21:35:00 DEBUG "data": { 21:35:00 DEBUG "resultType": "vector", 21:35:00 DEBUG "result": [ 21:35:00 DEBUG { 21:35:00 DEBUG "metric": { 21:35:00 DEBUG "instance": "10.142.0.100:9100" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG [...] 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG 21:35:00 DEBUG "instance": "10.142.0.99:9100" 21:35:00 DEBUG }, 21:35:00 DEBUG "value": [ 21:35:00 DEBUG 1686688500, 21:35:00 DEBUG "20313467.766666666" 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG ] 21:35:00 DEBUG } 21:35:00 DEBUG } 21:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688501" 21:35:01 INFO [http_cmd]: http status code OK 21:35:01 DEBUG --- status code --- 21:35:01 DEBUG http status code is 200 (expected 200) 21:35:01 DEBUG --- http response --- 21:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:35:01 DEBUG ----- output ----- 21:35:01 DEBUG { 21:35:01 DEBUG "status": "success", 21:35:01 DEBUG "data": { 21:35:01 DEBUG "resultType": "vector", 21:35:01 DEBUG "result": [ 21:35:01 DEBUG { 21:35:01 DEBUG "metric": { 21:35:01 DEBUG "pod": "ds-idrepo-2" 21:35:01 DEBUG }, 21:35:01 DEBUG "value": [ 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG [...] 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG "pod": "ds-idrepo-0" 21:35:01 DEBUG }, 21:35:01 DEBUG "value": [ 21:35:01 DEBUG 1686688501, 21:35:01 DEBUG "657.4" 21:35:01 DEBUG ] 21:35:01 DEBUG } 21:35:01 DEBUG ] 21:35:01 DEBUG } 21:35:01 DEBUG } 21:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688501" 21:35:01 INFO [http_cmd]: http status code OK 21:35:01 DEBUG --- status code --- 21:35:01 DEBUG http status code is 200 (expected 200) 21:35:01 DEBUG --- http response --- 21:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:35:01 DEBUG ----- output ----- 21:35:01 DEBUG { 21:35:01 DEBUG "status": "success", 21:35:01 DEBUG "data": { 21:35:01 DEBUG "resultType": "vector", 21:35:01 DEBUG "result": [ 21:35:01 DEBUG { 21:35:01 DEBUG "metric": { 21:35:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:35:01 DEBUG }, 21:35:01 DEBUG "value": [ 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG [...] 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:35:01 DEBUG }, 21:35:01 DEBUG "value": [ 21:35:01 DEBUG 1686688501, 21:35:01 DEBUG "0.2343647625364188" 21:35:01 DEBUG ] 21:35:01 DEBUG } 21:35:01 DEBUG ] 21:35:01 DEBUG } 21:35:01 DEBUG } 21:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688501" 21:35:01 INFO [http_cmd]: http status code OK 21:35:01 DEBUG --- status code --- 21:35:01 DEBUG http status code is 200 (expected 200) 21:35:01 DEBUG --- http response --- 21:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:35:01 DEBUG ----- output ----- 21:35:01 DEBUG { 21:35:01 DEBUG "status": "success", 21:35:01 DEBUG "data": { 21:35:01 DEBUG "resultType": "vector", 21:35:01 DEBUG "result": [ 21:35:01 DEBUG { 21:35:01 DEBUG "metric": { 21:35:01 DEBUG "pod": "ds-cts-2" 21:35:01 DEBUG }, 21:35:01 DEBUG "value": [ 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG [...] 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG 21:35:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:35:01 DEBUG }, 21:35:01 DEBUG "value": [ 21:35:01 DEBUG 1686688501, 21:35:01 DEBUG "0" 21:35:01 DEBUG ] 21:35:01 DEBUG } 21:35:01 DEBUG ] 21:35:01 DEBUG } 21:35:01 DEBUG } 21:35:15 INFO 21:35:15 INFO [loop_until]: kubectl --namespace=xlou top pods 21:35:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:35:15 INFO [loop_until]: OK (rc = 0) 21:35:15 DEBUG --- stdout --- 21:35:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 9m 5696Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 11m 416Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3285m 13800Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 15m 12908Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3143m 4825Mi idm-58dc667486-rgz9j 2160m 5088Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 204m 637Mi 21:35:15 DEBUG --- stderr --- 21:35:15 DEBUG 21:35:18 INFO 21:35:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:35:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:35:18 INFO [loop_until]: OK (rc = 0) 21:35:18 DEBUG --- stdout --- 21:35:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3037m 19% 5774Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6852Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 68m 0% 6808Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2611m 16% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 434m 2% 3149Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 59m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13703Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3182m 20% 14513Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 74m 0% 14578Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 265m 1% 1953Mi 3% 21:35:18 DEBUG --- stderr --- 21:35:18 DEBUG 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688560" 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG { 21:36:00 DEBUG { 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "data": { 21:36:00 DEBUG "data": { 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG "result": [ 21:36:00 DEBUG "result": [ 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG { 21:36:00 DEBUG { 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:36:00 DEBUG "pod": "ds-cts-0" 21:36:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:36:00 DEBUG "instance": "10.142.0.100:9100" 21:36:00 DEBUG }, 21:36:00 DEBUG }, 21:36:00 DEBUG }, 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG "value": [ 21:36:00 DEBUG "value": [ 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG [...] 21:36:00 DEBUG [...] 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:36:00 DEBUG "pod": "ds-idrepo-0" 21:36:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:36:00 DEBUG "instance": "10.142.0.99:9100" 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG }, 21:36:00 DEBUG }, 21:36:00 DEBUG }, 21:36:00 DEBUG }, 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:36:00 DEBUG "value": [ 21:36:00 DEBUG "value": [ 21:36:00 DEBUG "value": [ 21:36:00 DEBUG "value": [ 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG { 21:36:00 DEBUG "0" 21:36:00 DEBUG "600.1" 21:36:00 DEBUG "14642.74260189838" 21:36:00 DEBUG "0" 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG ] 21:36:00 DEBUG ] 21:36:00 DEBUG ] 21:36:00 DEBUG ] 21:36:00 DEBUG "data": { 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG ] 21:36:00 DEBUG ] 21:36:00 DEBUG ] 21:36:00 DEBUG ] 21:36:00 DEBUG "result": [ 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 DEBUG { 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "0" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "0" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "1.5705701525141507" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "ds-cts-0" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "ds-cts-2" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "0" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "0.2" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "ds-idrepo-2" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "ds-idrepo-0" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "600.0666666666666" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "0" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "ds-cts-0" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "ds-idrepo-0" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "0" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "13613809664" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "ds-idrepo-2" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "ds-idrepo-0" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "600.0666666666666" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "0" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "instance": "10.142.0.100:9100" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "instance": "10.142.0.99:9100" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "0.00043333333333445503" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "instance": "10.142.0.100:9100" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "instance": "10.142.0.99:9100" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "16957456.066666666" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688560" 21:36:00 INFO [http_cmd]: http status code OK 21:36:00 DEBUG --- status code --- 21:36:00 DEBUG http status code is 200 (expected 200) 21:36:00 DEBUG --- http response --- 21:36:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:36:00 DEBUG ----- output ----- 21:36:00 DEBUG { 21:36:00 DEBUG "status": "success", 21:36:00 DEBUG "data": { 21:36:00 DEBUG "resultType": "vector", 21:36:00 DEBUG "result": [ 21:36:00 DEBUG { 21:36:00 DEBUG "metric": { 21:36:00 DEBUG "instance": "10.142.0.100:9100" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG [...] 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG 21:36:00 DEBUG "instance": "10.142.0.99:9100" 21:36:00 DEBUG }, 21:36:00 DEBUG "value": [ 21:36:00 DEBUG 1686688560, 21:36:00 DEBUG "16955550.733333334" 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG ] 21:36:00 DEBUG } 21:36:00 DEBUG } 21:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688561" 21:36:01 INFO [http_cmd]: http status code OK 21:36:01 DEBUG --- status code --- 21:36:01 DEBUG http status code is 200 (expected 200) 21:36:01 DEBUG --- http response --- 21:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:36:01 DEBUG ----- output ----- 21:36:01 DEBUG { 21:36:01 DEBUG "status": "success", 21:36:01 DEBUG "data": { 21:36:01 DEBUG "resultType": "vector", 21:36:01 DEBUG "result": [ 21:36:01 DEBUG { 21:36:01 DEBUG "metric": { 21:36:01 DEBUG "pod": "ds-idrepo-2" 21:36:01 DEBUG }, 21:36:01 DEBUG "value": [ 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG [...] 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG "pod": "ds-idrepo-0" 21:36:01 DEBUG }, 21:36:01 DEBUG "value": [ 21:36:01 DEBUG 1686688561, 21:36:01 DEBUG "600.0333333333333" 21:36:01 DEBUG ] 21:36:01 DEBUG } 21:36:01 DEBUG ] 21:36:01 DEBUG } 21:36:01 DEBUG } 21:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688561" 21:36:01 INFO [http_cmd]: http status code OK 21:36:01 DEBUG --- status code --- 21:36:01 DEBUG http status code is 200 (expected 200) 21:36:01 DEBUG --- http response --- 21:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:36:01 DEBUG ----- output ----- 21:36:01 DEBUG { 21:36:01 DEBUG "status": "success", 21:36:01 DEBUG "data": { 21:36:01 DEBUG "resultType": "vector", 21:36:01 DEBUG "result": [ 21:36:01 DEBUG { 21:36:01 DEBUG "metric": { 21:36:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:36:01 DEBUG }, 21:36:01 DEBUG "value": [ 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG [...] 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:36:01 DEBUG }, 21:36:01 DEBUG "value": [ 21:36:01 DEBUG 1686688561, 21:36:01 DEBUG "0.21422161002421" 21:36:01 DEBUG ] 21:36:01 DEBUG } 21:36:01 DEBUG ] 21:36:01 DEBUG } 21:36:01 DEBUG } 21:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688561" 21:36:01 INFO [http_cmd]: http status code OK 21:36:01 DEBUG --- status code --- 21:36:01 DEBUG http status code is 200 (expected 200) 21:36:01 DEBUG --- http response --- 21:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:36:01 DEBUG ----- output ----- 21:36:01 DEBUG { 21:36:01 DEBUG "status": "success", 21:36:01 DEBUG "data": { 21:36:01 DEBUG "resultType": "vector", 21:36:01 DEBUG "result": [ 21:36:01 DEBUG { 21:36:01 DEBUG "metric": { 21:36:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:36:01 DEBUG }, 21:36:01 DEBUG "value": [ 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG [...] 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG 21:36:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:36:01 DEBUG }, 21:36:01 DEBUG "value": [ 21:36:01 DEBUG 1686688561, 21:36:01 DEBUG "0" 21:36:01 DEBUG ] 21:36:01 DEBUG } 21:36:01 DEBUG ] 21:36:01 DEBUG } 21:36:01 DEBUG } 21:36:15 INFO 21:36:15 INFO [loop_until]: kubectl --namespace=xlou top pods 21:36:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:36:15 INFO [loop_until]: OK (rc = 0) 21:36:15 DEBUG --- stdout --- 21:36:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5754Mi am-7849cf7bdb-ntqzs 10m 5696Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 423Mi ds-cts-2 8m 393Mi ds-idrepo-0 3279m 13803Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3095m 4827Mi idm-58dc667486-rgz9j 2643m 5088Mi lodemon-6455d5ff7c-zzndq 3m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 210m 636Mi 21:36:15 DEBUG --- stderr --- 21:36:15 DEBUG 21:36:18 INFO 21:36:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:36:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:36:18 INFO [loop_until]: OK (rc = 0) 21:36:18 DEBUG --- stdout --- 21:36:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3308m 20% 5777Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6811Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2702m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 445m 2% 3138Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1193Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13704Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3625m 22% 14491Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14582Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 269m 1% 1953Mi 3% 21:36:18 DEBUG --- stderr --- 21:36:18 DEBUG 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "ds-cts-0" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "ds-idrepo-0" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "597.0666666666666" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "0" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "ds-cts-2" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "50715.47735118735" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688620" 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "instance": "10.142.0.100:9100" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "instance": "10.142.0.99:9100" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "0.0003333333333333854" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "0" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "ds-cts-0" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "ds-cts-2" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "0" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "0" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "ds-cts-2" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "1.6109675107285737" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "0.2" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "ds-idrepo-2" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "ds-idrepo-0" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "597.0666666666666" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG { 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:37:00 DEBUG "data": { 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG { 21:37:00 DEBUG "result": [ 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG { 21:37:00 DEBUG "data": { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "pod": "ds-cts-0" 21:37:00 DEBUG "result": [ 21:37:00 DEBUG }, 21:37:00 DEBUG { 21:37:00 DEBUG "value": [ 21:37:00 DEBUG "metric": { 21:37:00 DEBUG 21:37:00 DEBUG "pod": "ds-cts-2" 21:37:00 DEBUG 21:37:00 DEBUG }, 21:37:00 DEBUG 21:37:00 DEBUG "value": [ 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG "pod": "ds-idrepo-0" 21:37:00 DEBUG 21:37:00 DEBUG }, 21:37:00 DEBUG 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:37:00 DEBUG "0" 21:37:00 DEBUG }, 21:37:00 DEBUG ] 21:37:00 DEBUG "value": [ 21:37:00 DEBUG } 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG ] 21:37:00 DEBUG "0" 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688620" 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "13617270784" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "ds-idrepo-2" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "ds-idrepo-0" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "597.0666666666666" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688620" 21:37:00 INFO [http_cmd]: http status code OK 21:37:00 DEBUG --- status code --- 21:37:00 DEBUG http status code is 200 (expected 200) 21:37:00 DEBUG --- http response --- 21:37:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:37:00 DEBUG ----- output ----- 21:37:00 DEBUG { 21:37:00 DEBUG "status": "success", 21:37:00 DEBUG "data": { 21:37:00 DEBUG "resultType": "vector", 21:37:00 DEBUG "result": [ 21:37:00 DEBUG { 21:37:00 DEBUG "metric": { 21:37:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG [...] 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG 21:37:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:37:00 DEBUG }, 21:37:00 DEBUG "value": [ 21:37:00 DEBUG 1686688620, 21:37:00 DEBUG "0" 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG ] 21:37:00 DEBUG } 21:37:00 DEBUG } 21:37:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688620" 21:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688621" 21:37:01 INFO [http_cmd]: http status code OK 21:37:01 DEBUG --- status code --- 21:37:01 DEBUG http status code is 200 (expected 200) 21:37:01 DEBUG --- http response --- 21:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:37:01 DEBUG ----- output ----- 21:37:01 DEBUG { 21:37:01 DEBUG "status": "success", 21:37:01 DEBUG "data": { 21:37:01 DEBUG "resultType": "vector", 21:37:01 DEBUG "result": [ 21:37:01 DEBUG { 21:37:01 DEBUG "metric": { 21:37:01 DEBUG "instance": "10.142.0.100:9100" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG [...] 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG "instance": "10.142.0.99:9100" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 1686688620, 21:37:01 DEBUG "18825663.666666668" 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG } 21:37:01 INFO [http_cmd]: http status code OK 21:37:01 DEBUG --- status code --- 21:37:01 DEBUG http status code is 200 (expected 200) 21:37:01 DEBUG --- http response --- 21:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:37:01 DEBUG ----- output ----- 21:37:01 DEBUG { 21:37:01 DEBUG "status": "success", 21:37:01 DEBUG "data": { 21:37:01 DEBUG "resultType": "vector", 21:37:01 DEBUG "result": [ 21:37:01 DEBUG { 21:37:01 DEBUG "metric": { 21:37:01 DEBUG "instance": "10.142.0.100:9100" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG [...] 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG "instance": "10.142.0.99:9100" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 1686688621, 21:37:01 DEBUG "0.0004999999999995453" 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG } 21:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688621" 21:37:01 INFO [http_cmd]: http status code OK 21:37:01 DEBUG --- status code --- 21:37:01 DEBUG http status code is 200 (expected 200) 21:37:01 DEBUG --- http response --- 21:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:37:01 DEBUG ----- output ----- 21:37:01 DEBUG { 21:37:01 DEBUG "status": "success", 21:37:01 DEBUG "data": { 21:37:01 DEBUG "resultType": "vector", 21:37:01 DEBUG "result": [ 21:37:01 DEBUG { 21:37:01 DEBUG "metric": { 21:37:01 DEBUG "instance": "10.142.0.100:9100" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG [...] 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG "instance": "10.142.0.99:9100" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 1686688621, 21:37:01 DEBUG "18827711.266666666" 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG } 21:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688621" 21:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688621" 21:37:01 INFO [http_cmd]: http status code OK 21:37:01 DEBUG --- status code --- 21:37:01 DEBUG http status code is 200 (expected 200) 21:37:01 DEBUG --- http response --- 21:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:37:01 DEBUG ----- output ----- 21:37:01 DEBUG { 21:37:01 DEBUG "status": "success", 21:37:01 DEBUG "data": { 21:37:01 DEBUG "resultType": "vector", 21:37:01 DEBUG "result": [ 21:37:01 DEBUG { 21:37:01 DEBUG "metric": { 21:37:01 DEBUG "pod": "ds-idrepo-2" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG [...] 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG "pod": "ds-idrepo-0" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 1686688621, 21:37:01 DEBUG "597.0666666666666" 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG } 21:37:01 INFO [http_cmd]: http status code OK 21:37:01 DEBUG --- status code --- 21:37:01 DEBUG http status code is 200 (expected 200) 21:37:01 DEBUG --- http response --- 21:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:37:01 DEBUG ----- output ----- 21:37:01 DEBUG { 21:37:01 DEBUG "status": "success", 21:37:01 DEBUG "data": { 21:37:01 DEBUG "resultType": "vector", 21:37:01 DEBUG "result": [ 21:37:01 DEBUG { 21:37:01 DEBUG "metric": { 21:37:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG [...] 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 1686688621, 21:37:01 DEBUG "0.22698182020722207" 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG } 21:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688621" 21:37:01 INFO [http_cmd]: http status code OK 21:37:01 DEBUG --- status code --- 21:37:01 DEBUG http status code is 200 (expected 200) 21:37:01 DEBUG --- http response --- 21:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:37:01 DEBUG ----- output ----- 21:37:01 DEBUG { 21:37:01 DEBUG "status": "success", 21:37:01 DEBUG "data": { 21:37:01 DEBUG "resultType": "vector", 21:37:01 DEBUG "result": [ 21:37:01 DEBUG { 21:37:01 DEBUG "metric": { 21:37:01 DEBUG "pod": "ds-cts-2" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG [...] 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG 21:37:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:37:01 DEBUG }, 21:37:01 DEBUG "value": [ 21:37:01 DEBUG 1686688621, 21:37:01 DEBUG "0" 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG ] 21:37:01 DEBUG } 21:37:01 DEBUG } 21:37:15 INFO 21:37:15 INFO [loop_until]: kubectl --namespace=xlou top pods 21:37:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:37:15 INFO [loop_until]: OK (rc = 0) 21:37:15 DEBUG --- stdout --- 21:37:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 10m 5696Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3314m 13800Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 13m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3206m 4829Mi idm-58dc667486-rgz9j 2941m 5088Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 211m 637Mi 21:37:15 DEBUG --- stderr --- 21:37:15 DEBUG 21:37:18 INFO 21:37:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:37:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:37:18 INFO [loop_until]: OK (rc = 0) 21:37:18 DEBUG --- stdout --- 21:37:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3032m 19% 5778Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 3108m 19% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 77m 0% 6959Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 453m 2% 3140Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1163Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1194Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13704Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3549m 22% 14495Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14580Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 286m 1% 1952Mi 3% 21:37:18 DEBUG --- stderr --- 21:37:18 DEBUG 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "0" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:38:00 DEBUG { 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG { 21:38:00 DEBUG "data": { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "result": [ 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG { 21:38:00 DEBUG "result": [ 21:38:00 DEBUG "metric": { 21:38:00 DEBUG { 21:38:00 DEBUG "pod": "ds-cts-0" 21:38:00 DEBUG "metric": { 21:38:00 DEBUG }, 21:38:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:38:00 DEBUG "value": [ 21:38:00 DEBUG }, 21:38:00 DEBUG 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "ds-idrepo-0" 21:38:00 DEBUG 21:38:00 DEBUG }, 21:38:00 DEBUG "pod": "ds-idrepo-0" 21:38:00 DEBUG "value": [ 21:38:00 DEBUG }, 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG "589.4666666666667" 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG ] 21:38:00 DEBUG "432997084.45673466" 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688680" 21:38:00 DEBUG ] 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "0" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "instance": "10.142.0.100:9100" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "instance": "10.142.0.99:9100" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "0" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "pod": "ds-cts-0" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "ds-cts-2" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "0" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "ds-idrepo-0" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "3.4397046469352426" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "0.2" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "0" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "pod": "ds-idrepo-2" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "ds-idrepo-0" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "589.4666666666667" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG { 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG "data": { 21:38:00 DEBUG { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG "data": { 21:38:00 DEBUG { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "result": [ 21:38:00 DEBUG "pod": "ds-cts-0" 21:38:00 DEBUG { 21:38:00 DEBUG }, 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "value": [ 21:38:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:38:00 DEBUG 21:38:00 DEBUG }, 21:38:00 DEBUG 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "ds-idrepo-0" 21:38:00 DEBUG 21:38:00 DEBUG }, 21:38:00 DEBUG 21:38:00 DEBUG "value": [ 21:38:00 DEBUG "pod": "ds-idrepo-0" 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG }, 21:38:00 DEBUG "0" 21:38:00 DEBUG "value": [ 21:38:00 DEBUG ] 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG } 21:38:00 DEBUG "6980.095633137458" 21:38:00 DEBUG ] 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688680" 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "13615316992" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "pod": "ds-idrepo-2" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "ds-idrepo-0" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "589.4333333333333" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688680" 21:38:00 INFO [http_cmd]: http status code OK 21:38:00 DEBUG --- status code --- 21:38:00 DEBUG http status code is 200 (expected 200) 21:38:00 DEBUG --- http response --- 21:38:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:38:00 DEBUG ----- output ----- 21:38:00 DEBUG { 21:38:00 DEBUG "status": "success", 21:38:00 DEBUG "data": { 21:38:00 DEBUG "resultType": "vector", 21:38:00 DEBUG "result": [ 21:38:00 DEBUG { 21:38:00 DEBUG "metric": { 21:38:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG [...] 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG 21:38:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:38:00 DEBUG }, 21:38:00 DEBUG "value": [ 21:38:00 DEBUG 1686688680, 21:38:00 DEBUG "0" 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG ] 21:38:00 DEBUG } 21:38:00 DEBUG } 21:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688681" 21:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688681" 21:38:01 INFO [http_cmd]: http status code OK 21:38:01 DEBUG --- status code --- 21:38:01 DEBUG http status code is 200 (expected 200) 21:38:01 DEBUG --- http response --- 21:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:38:01 DEBUG ----- output ----- 21:38:01 DEBUG { 21:38:01 DEBUG "status": "success", 21:38:01 DEBUG "data": { 21:38:01 DEBUG "resultType": "vector", 21:38:01 DEBUG "result": [ 21:38:01 DEBUG { 21:38:01 DEBUG "metric": { 21:38:01 DEBUG "instance": "10.142.0.100:9100" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG [...] 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG "instance": "10.142.0.99:9100" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 1686688681, 21:38:01 DEBUG "20327226.233333334" 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG } 21:38:01 INFO [http_cmd]: http status code OK 21:38:01 DEBUG --- status code --- 21:38:01 DEBUG http status code is 200 (expected 200) 21:38:01 DEBUG --- http response --- 21:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:38:01 DEBUG ----- output ----- 21:38:01 DEBUG { 21:38:01 DEBUG "status": "success", 21:38:01 DEBUG "data": { 21:38:01 DEBUG "resultType": "vector", 21:38:01 DEBUG "result": [ 21:38:01 DEBUG { 21:38:01 DEBUG "metric": { 21:38:01 DEBUG "instance": "10.142.0.100:9100" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG [...] 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG "instance": "10.142.0.99:9100" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 1686688681, 21:38:01 DEBUG "20329122.099999998" 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG } 21:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688681" 21:38:01 INFO [http_cmd]: http status code OK 21:38:01 DEBUG --- status code --- 21:38:01 DEBUG http status code is 200 (expected 200) 21:38:01 DEBUG --- http response --- 21:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:38:01 DEBUG ----- output ----- 21:38:01 DEBUG { 21:38:01 DEBUG "status": "success", 21:38:01 DEBUG "data": { 21:38:01 DEBUG "resultType": "vector", 21:38:01 DEBUG "result": [ 21:38:01 DEBUG { 21:38:01 DEBUG "metric": { 21:38:01 DEBUG "instance": "10.142.0.100:9100" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG [...] 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG "instance": "10.142.0.99:9100" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 1686688681, 21:38:01 DEBUG "0.00036666666666747006" 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG } 21:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688681" 21:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688681" 21:38:01 INFO [http_cmd]: http status code OK 21:38:01 DEBUG --- status code --- 21:38:01 DEBUG http status code is 200 (expected 200) 21:38:01 DEBUG --- http response --- 21:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:38:01 DEBUG ----- output ----- 21:38:01 DEBUG { 21:38:01 DEBUG "status": "success", 21:38:01 DEBUG "data": { 21:38:01 DEBUG "resultType": "vector", 21:38:01 DEBUG "result": [ 21:38:01 DEBUG { 21:38:01 DEBUG "metric": { 21:38:01 DEBUG "pod": "ds-idrepo-2" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG [...] 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG "pod": "ds-idrepo-0" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 1686688681, 21:38:01 DEBUG "589.4333333333333" 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG } 21:38:01 INFO [http_cmd]: http status code OK 21:38:01 DEBUG --- status code --- 21:38:01 DEBUG http status code is 200 (expected 200) 21:38:01 DEBUG --- http response --- 21:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:38:01 DEBUG ----- output ----- 21:38:01 DEBUG { 21:38:01 DEBUG "status": "success", 21:38:01 DEBUG "data": { 21:38:01 DEBUG "resultType": "vector", 21:38:01 DEBUG "result": [ 21:38:01 DEBUG { 21:38:01 DEBUG "metric": { 21:38:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG [...] 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 1686688681, 21:38:01 DEBUG "0.2364142515264538" 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG } 21:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688681" 21:38:01 INFO [http_cmd]: http status code OK 21:38:01 DEBUG --- status code --- 21:38:01 DEBUG http status code is 200 (expected 200) 21:38:01 DEBUG --- http response --- 21:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:38:01 DEBUG ----- output ----- 21:38:01 DEBUG { 21:38:01 DEBUG "status": "success", 21:38:01 DEBUG "data": { 21:38:01 DEBUG "resultType": "vector", 21:38:01 DEBUG "result": [ 21:38:01 DEBUG { 21:38:01 DEBUG "metric": { 21:38:01 DEBUG "pod": "ds-cts-2" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG [...] 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG 21:38:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:38:01 DEBUG }, 21:38:01 DEBUG "value": [ 21:38:01 DEBUG 1686688681, 21:38:01 DEBUG "0" 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG ] 21:38:01 DEBUG } 21:38:01 DEBUG } 21:38:15 INFO 21:38:15 INFO [loop_until]: kubectl --namespace=xlou top pods 21:38:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:38:15 INFO [loop_until]: OK (rc = 0) 21:38:15 DEBUG --- stdout --- 21:38:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3415m 13798Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3144m 4831Mi idm-58dc667486-rgz9j 2820m 5089Mi lodemon-6455d5ff7c-zzndq 4m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 211m 637Mi 21:38:15 DEBUG --- stderr --- 21:38:15 DEBUG 21:38:18 INFO 21:38:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:38:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:38:18 INFO [loop_until]: OK (rc = 0) 21:38:18 DEBUG --- stdout --- 21:38:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2995m 18% 5784Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 6819Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2993m 18% 6074Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 69m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 450m 2% 3133Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1162Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1193Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13705Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3546m 22% 14488Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14582Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 282m 1% 1953Mi 3% 21:38:18 DEBUG --- stderr --- 21:38:18 DEBUG 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "0" 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688740" 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:39:00 DEBUG { 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG { 21:39:00 DEBUG { 21:39:00 DEBUG "data": { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "data": { 21:39:00 DEBUG "result": [ 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG { 21:39:00 DEBUG "result": [ 21:39:00 DEBUG "result": [ 21:39:00 DEBUG "metric": { 21:39:00 DEBUG { 21:39:00 DEBUG { 21:39:00 DEBUG "pod": "ds-cts-0" 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG }, 21:39:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:39:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:39:00 DEBUG "value": [ 21:39:00 DEBUG }, 21:39:00 DEBUG }, 21:39:00 DEBUG 21:39:00 DEBUG "value": [ 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "ds-idrepo-0" 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG }, 21:39:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:39:00 DEBUG "pod": "ds-cts-0" 21:39:00 DEBUG "value": [ 21:39:00 DEBUG }, 21:39:00 DEBUG }, 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG "value": [ 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG "634.3666666666667" 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG ] 21:39:00 DEBUG "0" 21:39:00 DEBUG "0" 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG ] 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG ] 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 DEBUG { 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG } 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG "data": { 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "data": { 21:39:00 DEBUG { 21:39:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG }, 21:39:00 DEBUG "result": [ 21:39:00 DEBUG "data": { 21:39:00 DEBUG "value": [ 21:39:00 DEBUG { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "result": [ 21:39:00 DEBUG 21:39:00 DEBUG "instance": "10.142.0.100:9100" 21:39:00 DEBUG { 21:39:00 DEBUG 21:39:00 DEBUG }, 21:39:00 DEBUG "metric": { 21:39:00 DEBUG [...] 21:39:00 DEBUG "value": [ 21:39:00 DEBUG "pod": "ds-cts-0" 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG }, 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG }, 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "0.2" 21:39:00 DEBUG "instance": "10.142.0.99:9100" 21:39:00 DEBUG 21:39:00 DEBUG ] 21:39:00 DEBUG }, 21:39:00 DEBUG 21:39:00 DEBUG } 21:39:00 DEBUG "value": [ 21:39:00 DEBUG "pod": "ds-cts-2" 21:39:00 DEBUG ] 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG }, 21:39:00 DEBUG } 21:39:00 DEBUG "0" 21:39:00 DEBUG "value": [ 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG } 21:39:00 DEBUG "0" 21:39:00 DEBUG ] 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "ds-cts-0" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "0" 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "0" 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "pod": "ds-idrepo-2" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "ds-idrepo-0" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "634.3333333333334" 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "pod": "ds-cts-0" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "ds-idrepo-0" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "0" 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "0" 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688740" 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "13615513600" 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "pod": "ds-idrepo-2" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "ds-idrepo-0" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "634.3666666666667" 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688740" 21:39:00 INFO [http_cmd]: http status code OK 21:39:00 DEBUG --- status code --- 21:39:00 DEBUG http status code is 200 (expected 200) 21:39:00 DEBUG --- http response --- 21:39:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:39:00 DEBUG ----- output ----- 21:39:00 DEBUG { 21:39:00 DEBUG "status": "success", 21:39:00 DEBUG "data": { 21:39:00 DEBUG "resultType": "vector", 21:39:00 DEBUG "result": [ 21:39:00 DEBUG { 21:39:00 DEBUG "metric": { 21:39:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG [...] 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG 21:39:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:39:00 DEBUG }, 21:39:00 DEBUG "value": [ 21:39:00 DEBUG 1686688740, 21:39:00 DEBUG "0" 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG ] 21:39:00 DEBUG } 21:39:00 DEBUG } 21:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688741" 21:39:01 INFO [http_cmd]: http status code OK 21:39:01 DEBUG --- status code --- 21:39:01 DEBUG http status code is 200 (expected 200) 21:39:01 DEBUG --- http response --- 21:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:39:01 DEBUG ----- output ----- 21:39:01 DEBUG { 21:39:01 DEBUG "status": "success", 21:39:01 DEBUG "data": { 21:39:01 DEBUG "resultType": "vector", 21:39:01 DEBUG "result": [ 21:39:01 DEBUG { 21:39:01 DEBUG "metric": { 21:39:01 DEBUG "instance": "10.142.0.100:9100" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG [...] 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG "instance": "10.142.0.99:9100" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 1686688741, 21:39:01 DEBUG "19669859.533333335" 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG } 21:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688741" 21:39:01 INFO [http_cmd]: http status code OK 21:39:01 DEBUG --- status code --- 21:39:01 DEBUG http status code is 200 (expected 200) 21:39:01 DEBUG --- http response --- 21:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:39:01 DEBUG ----- output ----- 21:39:01 DEBUG { 21:39:01 DEBUG "status": "success", 21:39:01 DEBUG "data": { 21:39:01 DEBUG "resultType": "vector", 21:39:01 DEBUG "result": [ 21:39:01 DEBUG { 21:39:01 DEBUG "metric": { 21:39:01 DEBUG "instance": "10.142.0.100:9100" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG [...] 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG "instance": "10.142.0.99:9100" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 1686688741, 21:39:01 DEBUG "19671778.833333332" 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG } 21:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688741" 21:39:01 INFO [http_cmd]: http status code OK 21:39:01 DEBUG --- status code --- 21:39:01 DEBUG http status code is 200 (expected 200) 21:39:01 DEBUG --- http response --- 21:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:39:01 DEBUG ----- output ----- 21:39:01 DEBUG { 21:39:01 DEBUG "status": "success", 21:39:01 DEBUG "data": { 21:39:01 DEBUG "resultType": "vector", 21:39:01 DEBUG "result": [ 21:39:01 DEBUG { 21:39:01 DEBUG "metric": { 21:39:01 DEBUG "instance": "10.142.0.100:9100" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG [...] 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG "instance": "10.142.0.99:9100" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 1686688741, 21:39:01 DEBUG "0.00040000000000001514" 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG } 21:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688741" 21:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688741" 21:39:01 INFO [http_cmd]: http status code OK 21:39:01 DEBUG --- status code --- 21:39:01 DEBUG http status code is 200 (expected 200) 21:39:01 DEBUG --- http response --- 21:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:39:01 DEBUG ----- output ----- 21:39:01 DEBUG { 21:39:01 DEBUG "status": "success", 21:39:01 DEBUG "data": { 21:39:01 DEBUG "resultType": "vector", 21:39:01 DEBUG "result": [ 21:39:01 DEBUG { 21:39:01 DEBUG "metric": { 21:39:01 DEBUG "pod": "ds-idrepo-2" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG [...] 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG "pod": "ds-idrepo-0" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 1686688741, 21:39:01 DEBUG "634.3333333333334" 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG } 21:39:01 INFO [http_cmd]: http status code OK 21:39:01 DEBUG --- status code --- 21:39:01 DEBUG http status code is 200 (expected 200) 21:39:01 DEBUG --- http response --- 21:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:39:01 DEBUG ----- output ----- 21:39:01 DEBUG { 21:39:01 DEBUG "status": "success", 21:39:01 DEBUG "data": { 21:39:01 DEBUG "resultType": "vector", 21:39:01 DEBUG "result": [ 21:39:01 DEBUG { 21:39:01 DEBUG "metric": { 21:39:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG [...] 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 1686688741, 21:39:01 DEBUG "0.23909465626581336" 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG } 21:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688741" 21:39:01 INFO [http_cmd]: http status code OK 21:39:01 DEBUG --- status code --- 21:39:01 DEBUG http status code is 200 (expected 200) 21:39:01 DEBUG --- http response --- 21:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:39:01 DEBUG ----- output ----- 21:39:01 DEBUG { 21:39:01 DEBUG "status": "success", 21:39:01 DEBUG "data": { 21:39:01 DEBUG "resultType": "vector", 21:39:01 DEBUG "result": [ 21:39:01 DEBUG { 21:39:01 DEBUG "metric": { 21:39:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG [...] 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG 21:39:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:39:01 DEBUG }, 21:39:01 DEBUG "value": [ 21:39:01 DEBUG 1686688741, 21:39:01 DEBUG "0" 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG ] 21:39:01 DEBUG } 21:39:01 DEBUG } 21:39:15 INFO 21:39:15 INFO [loop_until]: kubectl --namespace=xlou top pods 21:39:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:39:15 INFO [loop_until]: OK (rc = 0) 21:39:15 DEBUG --- stdout --- 21:39:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3598m 13808Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3256m 4833Mi idm-58dc667486-rgz9j 2804m 5089Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 223m 638Mi 21:39:15 DEBUG --- stderr --- 21:39:15 DEBUG 21:39:18 INFO 21:39:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:39:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:39:18 INFO [loop_until]: OK (rc = 0) 21:39:18 DEBUG --- stdout --- 21:39:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3344m 21% 5784Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 72m 0% 6868Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 69m 0% 6816Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2934m 18% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 454m 2% 3141Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 57m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1194Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 66m 0% 13706Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3582m 22% 14513Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 69m 0% 14585Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 275m 1% 1965Mi 3% 21:39:18 DEBUG --- stderr --- 21:39:18 DEBUG 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "ds-cts-2" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:40:00 DEBUG "pod": "ds-cts-0" 21:40:00 DEBUG }, 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:40:00 DEBUG "pod": "ds-idrepo-0" 21:40:00 DEBUG }, 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0.2" 21:40:00 DEBUG "638.2" 21:40:00 DEBUG ] 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "instance": "10.142.0.100:9100" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "instance": "10.142.0.99:9100" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "ds-cts-0" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "ds-cts-2" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "ds-cts-2" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0.0000335155394905809" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "ds-idrepo-2" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "ds-idrepo-0" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "638.1333333333333" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "ds-cts-0" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "ds-idrepo-0" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "pod": "ds-cts-2" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "0" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688800" 21:40:00 INFO [http_cmd]: http status code OK 21:40:00 DEBUG --- status code --- 21:40:00 DEBUG http status code is 200 (expected 200) 21:40:00 DEBUG --- http response --- 21:40:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:40:00 DEBUG ----- output ----- 21:40:00 DEBUG { 21:40:00 DEBUG "status": "success", 21:40:00 DEBUG "data": { 21:40:00 DEBUG "resultType": "vector", 21:40:00 DEBUG "result": [ 21:40:00 DEBUG { 21:40:00 DEBUG "metric": { 21:40:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG [...] 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG 21:40:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:40:00 DEBUG }, 21:40:00 DEBUG "value": [ 21:40:00 DEBUG 1686688800, 21:40:00 DEBUG "13616472064" 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG ] 21:40:00 DEBUG } 21:40:00 DEBUG } 21:40:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688800" 21:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688801" 21:40:01 INFO [http_cmd]: http status code OK 21:40:01 DEBUG --- status code --- 21:40:01 DEBUG http status code is 200 (expected 200) 21:40:01 DEBUG --- http response --- 21:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:40:01 DEBUG ----- output ----- 21:40:01 DEBUG { 21:40:01 DEBUG "status": "success", 21:40:01 DEBUG "data": { 21:40:01 DEBUG "resultType": "vector", 21:40:01 DEBUG "result": [ 21:40:01 DEBUG { 21:40:01 DEBUG "metric": { 21:40:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG [...] 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 1686688800, 21:40:01 DEBUG "0" 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG } 21:40:01 INFO [http_cmd]: http status code OK 21:40:01 DEBUG --- status code --- 21:40:01 DEBUG http status code is 200 (expected 200) 21:40:01 DEBUG --- http response --- 21:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:40:01 DEBUG ----- output ----- 21:40:01 DEBUG { 21:40:01 DEBUG "status": "success", 21:40:01 DEBUG "data": { 21:40:01 DEBUG "resultType": "vector", 21:40:01 DEBUG "result": [ 21:40:01 DEBUG { 21:40:01 DEBUG "metric": { 21:40:01 DEBUG "pod": "ds-idrepo-2" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG [...] 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG "pod": "ds-idrepo-0" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 1686688801, 21:40:01 DEBUG "638.1" 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG } 21:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688801" 21:40:01 INFO [http_cmd]: http status code OK 21:40:01 DEBUG --- status code --- 21:40:01 DEBUG http status code is 200 (expected 200) 21:40:01 DEBUG --- http response --- 21:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:40:01 DEBUG ----- output ----- 21:40:01 DEBUG { 21:40:01 DEBUG "status": "success", 21:40:01 DEBUG "data": { 21:40:01 DEBUG "resultType": "vector", 21:40:01 DEBUG "result": [ 21:40:01 DEBUG { 21:40:01 DEBUG "metric": { 21:40:01 DEBUG "instance": "10.142.0.100:9100" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG [...] 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG "instance": "10.142.0.99:9100" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 1686688801, 21:40:01 DEBUG "21466569.733333334" 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG } 21:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688801" 21:40:01 INFO [http_cmd]: http status code OK 21:40:01 DEBUG --- status code --- 21:40:01 DEBUG http status code is 200 (expected 200) 21:40:01 DEBUG --- http response --- 21:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:40:01 DEBUG ----- output ----- 21:40:01 DEBUG { 21:40:01 DEBUG "status": "success", 21:40:01 DEBUG "data": { 21:40:01 DEBUG "resultType": "vector", 21:40:01 DEBUG "result": [ 21:40:01 DEBUG { 21:40:01 DEBUG "metric": { 21:40:01 DEBUG "instance": "10.142.0.100:9100" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG [...] 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG "instance": "10.142.0.99:9100" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 1686688801, 21:40:01 DEBUG "21468854.866666667" 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG } 21:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688801" 21:40:01 INFO [http_cmd]: http status code OK 21:40:01 DEBUG --- status code --- 21:40:01 DEBUG http status code is 200 (expected 200) 21:40:01 DEBUG --- http response --- 21:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:40:01 DEBUG ----- output ----- 21:40:01 DEBUG { 21:40:01 DEBUG "status": "success", 21:40:01 DEBUG "data": { 21:40:01 DEBUG "resultType": "vector", 21:40:01 DEBUG "result": [ 21:40:01 DEBUG { 21:40:01 DEBUG "metric": { 21:40:01 DEBUG "instance": "10.142.0.100:9100" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG [...] 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG "instance": "10.142.0.99:9100" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 1686688801, 21:40:01 DEBUG "0.00023333333333350008" 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG } 21:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688801" 21:40:01 INFO [http_cmd]: http status code OK 21:40:01 DEBUG --- status code --- 21:40:01 DEBUG http status code is 200 (expected 200) 21:40:01 DEBUG --- http response --- 21:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:40:01 DEBUG ----- output ----- 21:40:01 DEBUG { 21:40:01 DEBUG "status": "success", 21:40:01 DEBUG "data": { 21:40:01 DEBUG "resultType": "vector", 21:40:01 DEBUG "result": [ 21:40:01 DEBUG { 21:40:01 DEBUG "metric": { 21:40:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG [...] 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 1686688801, 21:40:01 DEBUG "0.23051604805328954" 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG } 21:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688801" 21:40:01 INFO [http_cmd]: http status code OK 21:40:01 DEBUG --- status code --- 21:40:01 DEBUG http status code is 200 (expected 200) 21:40:01 DEBUG --- http response --- 21:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:40:01 DEBUG ----- output ----- 21:40:01 DEBUG { 21:40:01 DEBUG "status": "success", 21:40:01 DEBUG "data": { 21:40:01 DEBUG "resultType": "vector", 21:40:01 DEBUG "result": [ 21:40:01 DEBUG { 21:40:01 DEBUG "metric": { 21:40:01 DEBUG "pod": "ds-idrepo-2" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG [...] 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG "pod": "ds-idrepo-0" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 1686688801, 21:40:01 DEBUG "638.1333333333333" 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG } 21:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688801" 21:40:01 INFO [http_cmd]: http status code OK 21:40:01 DEBUG --- status code --- 21:40:01 DEBUG http status code is 200 (expected 200) 21:40:01 DEBUG --- http response --- 21:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:40:01 DEBUG ----- output ----- 21:40:01 DEBUG { 21:40:01 DEBUG "status": "success", 21:40:01 DEBUG "data": { 21:40:01 DEBUG "resultType": "vector", 21:40:01 DEBUG "result": [ 21:40:01 DEBUG { 21:40:01 DEBUG "metric": { 21:40:01 DEBUG "pod": "ds-cts-2" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG [...] 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG 21:40:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:40:01 DEBUG }, 21:40:01 DEBUG "value": [ 21:40:01 DEBUG 1686688801, 21:40:01 DEBUG "0" 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG ] 21:40:01 DEBUG } 21:40:01 DEBUG } 21:40:15 INFO 21:40:15 INFO [loop_until]: kubectl --namespace=xlou top pods 21:40:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:40:15 INFO [loop_until]: OK (rc = 0) 21:40:15 DEBUG --- stdout --- 21:40:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 422Mi ds-cts-2 9m 392Mi ds-idrepo-0 3435m 13802Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 17m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2887m 4835Mi idm-58dc667486-rgz9j 2734m 5089Mi lodemon-6455d5ff7c-zzndq 4m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 195m 638Mi 21:40:15 DEBUG --- stderr --- 21:40:15 DEBUG 21:40:18 INFO 21:40:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:40:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:40:18 INFO [loop_until]: OK (rc = 0) 21:40:18 DEBUG --- stdout --- 21:40:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3003m 18% 5787Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 72m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2789m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 444m 2% 3163Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13702Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3384m 21% 14496Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14585Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 259m 1% 1953Mi 3% 21:40:18 DEBUG --- stderr --- 21:40:18 DEBUG 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG { 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG "data": { 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG [...] 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG "0" 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG } 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG { 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG "data": { 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "pod": "ds-cts-0" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG [...] 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "pod": "ds-idrepo-0" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG "606.1333333333333" 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG } 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG { 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG "data": { 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "pod": "ds-cts-0" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG [...] 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "pod": "ds-cts-2" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG "0" 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG } 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG { 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG "data": { 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG [...] 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG "0" 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG } 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG { 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG "data": { 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG [...] 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG "0.2" 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG } 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688860" 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG { 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG "data": { 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:41:00 DEBUG }, 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG "value": [ 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:41:00 DEBUG [...] 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG 21:41:00 DEBUG { 21:41:00 DEBUG 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG 21:41:00 DEBUG "data": { 21:41:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG }, 21:41:00 DEBUG "result": [ 21:41:00 DEBUG "value": [ 21:41:00 DEBUG { 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "0" 21:41:00 DEBUG "instance": "10.142.0.100:9100" 21:41:00 DEBUG ] 21:41:00 DEBUG }, 21:41:00 DEBUG } 21:41:00 DEBUG "value": [ 21:41:00 DEBUG ] 21:41:00 DEBUG 21:41:00 DEBUG } 21:41:00 DEBUG 21:41:00 DEBUG } 21:41:00 DEBUG 21:41:00 DEBUG [...] 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "instance": "10.142.0.99:9100" 21:41:00 DEBUG }, 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG "value": [ 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG "0.0003333333333333854" 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG ] 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:41:00 DEBUG } 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG ] 21:41:00 DEBUG { 21:41:00 DEBUG } 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG } 21:41:00 DEBUG "data": { 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG [...] 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG "0" 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG } 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG { 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG "data": { 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG { 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "data": { 21:41:00 DEBUG "pod": "ds-idrepo-2" 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG }, 21:41:00 DEBUG "result": [ 21:41:00 DEBUG "value": [ 21:41:00 DEBUG { 21:41:00 DEBUG 21:41:00 DEBUG "metric": { 21:41:00 DEBUG 21:41:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:41:00 DEBUG 21:41:00 DEBUG }, 21:41:00 DEBUG [...] 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "pod": "ds-idrepo-0" 21:41:00 DEBUG [...] 21:41:00 DEBUG }, 21:41:00 DEBUG 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG 21:41:00 DEBUG "606.1333333333333" 21:41:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:41:00 DEBUG ] 21:41:00 DEBUG }, 21:41:00 DEBUG } 21:41:00 DEBUG "value": [ 21:41:00 DEBUG ] 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG } 21:41:00 DEBUG "0" 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG } 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG { 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG "data": { 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "pod": "ds-cts-0" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG [...] 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "pod": "ds-idrepo-0" 21:41:00 DEBUG }, 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG "0" 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG } 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688860" 21:41:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688860" 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:41:00 INFO [http_cmd]: http status code OK 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG --- status code --- 21:41:00 DEBUG { 21:41:00 DEBUG http status code is 200 (expected 200) 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG --- http response --- 21:41:00 DEBUG "data": { 21:41:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG ----- output ----- 21:41:00 DEBUG "result": [ 21:41:00 DEBUG { 21:41:00 DEBUG { 21:41:00 DEBUG "status": "success", 21:41:00 DEBUG "metric": { 21:41:00 DEBUG "data": { 21:41:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:41:00 DEBUG "resultType": "vector", 21:41:00 DEBUG }, 21:41:00 DEBUG "result": [ 21:41:00 DEBUG "value": [ 21:41:00 DEBUG { 21:41:00 DEBUG 21:41:00 DEBUG "metric": { 21:41:00 DEBUG 21:41:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:41:00 DEBUG 21:41:00 DEBUG }, 21:41:00 DEBUG [...] 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG 21:41:00 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 21:41:00 DEBUG [...] 21:41:00 DEBUG }, 21:41:00 DEBUG 21:41:00 DEBUG "value": [ 21:41:00 DEBUG 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG 21:41:00 DEBUG "0" 21:41:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:41:00 DEBUG ] 21:41:00 DEBUG }, 21:41:00 DEBUG } 21:41:00 DEBUG "value": [ 21:41:00 DEBUG ] 21:41:00 DEBUG 1686688860, 21:41:00 DEBUG } 21:41:00 DEBUG "13615898624" 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG ] 21:41:00 DEBUG } 21:41:00 DEBUG } 21:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688861" 21:41:01 INFO [http_cmd]: http status code OK 21:41:01 DEBUG --- status code --- 21:41:01 DEBUG http status code is 200 (expected 200) 21:41:01 DEBUG --- http response --- 21:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:41:01 DEBUG ----- output ----- 21:41:01 DEBUG { 21:41:01 DEBUG "status": "success", 21:41:01 DEBUG "data": { 21:41:01 DEBUG "resultType": "vector", 21:41:01 DEBUG "result": [ 21:41:01 DEBUG { 21:41:01 DEBUG "metric": { 21:41:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG [...] 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 1686688861, 21:41:01 DEBUG "0" 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG } 21:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688861" 21:41:01 INFO [http_cmd]: http status code OK 21:41:01 DEBUG --- status code --- 21:41:01 DEBUG http status code is 200 (expected 200) 21:41:01 DEBUG --- http response --- 21:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:41:01 DEBUG ----- output ----- 21:41:01 DEBUG { 21:41:01 DEBUG "status": "success", 21:41:01 DEBUG "data": { 21:41:01 DEBUG "resultType": "vector", 21:41:01 DEBUG "result": [ 21:41:01 DEBUG { 21:41:01 DEBUG "metric": { 21:41:01 DEBUG "pod": "ds-idrepo-2" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG [...] 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG "pod": "ds-idrepo-0" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 1686688861, 21:41:01 DEBUG "606.1333333333333" 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG } 21:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688861" 21:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688861" 21:41:01 INFO [http_cmd]: http status code OK 21:41:01 INFO [http_cmd]: http status code OK 21:41:01 DEBUG --- status code --- 21:41:01 DEBUG --- status code --- 21:41:01 DEBUG http status code is 200 (expected 200) 21:41:01 DEBUG http status code is 200 (expected 200) 21:41:01 DEBUG --- http response --- 21:41:01 DEBUG --- http response --- 21:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:41:01 DEBUG ----- output ----- 21:41:01 DEBUG { 21:41:01 DEBUG "status": "success", 21:41:01 DEBUG "data": { 21:41:01 DEBUG "resultType": "vector", 21:41:01 DEBUG "result": [ 21:41:01 DEBUG { 21:41:01 DEBUG "metric": { 21:41:01 DEBUG "instance": "10.142.0.100:9100" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:41:01 DEBUG 21:41:01 DEBUG ----- output ----- 21:41:01 DEBUG 21:41:01 DEBUG { 21:41:01 DEBUG 21:41:01 DEBUG "status": "success", 21:41:01 DEBUG [...] 21:41:01 DEBUG "data": { 21:41:01 DEBUG 21:41:01 DEBUG "resultType": "vector", 21:41:01 DEBUG 21:41:01 DEBUG "result": [ 21:41:01 DEBUG 21:41:01 DEBUG { 21:41:01 DEBUG "instance": "10.142.0.99:9100" 21:41:01 DEBUG "metric": { 21:41:01 DEBUG }, 21:41:01 DEBUG "instance": "10.142.0.100:9100" 21:41:01 DEBUG "value": [ 21:41:01 DEBUG }, 21:41:01 DEBUG 1686688861, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG "19843545.366666667" 21:41:01 DEBUG 21:41:01 DEBUG ] 21:41:01 DEBUG 21:41:01 DEBUG } 21:41:01 DEBUG 21:41:01 DEBUG ] 21:41:01 DEBUG [...] 21:41:01 DEBUG } 21:41:01 DEBUG 21:41:01 DEBUG } 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG "instance": "10.142.0.99:9100" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 1686688861, 21:41:01 DEBUG "19843731.233333334" 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG } 21:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688861" 21:41:01 INFO [http_cmd]: http status code OK 21:41:01 DEBUG --- status code --- 21:41:01 DEBUG http status code is 200 (expected 200) 21:41:01 DEBUG --- http response --- 21:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:41:01 DEBUG ----- output ----- 21:41:01 DEBUG { 21:41:01 DEBUG "status": "success", 21:41:01 DEBUG "data": { 21:41:01 DEBUG "resultType": "vector", 21:41:01 DEBUG "result": [ 21:41:01 DEBUG { 21:41:01 DEBUG "metric": { 21:41:01 DEBUG "instance": "10.142.0.100:9100" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG [...] 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG "instance": "10.142.0.99:9100" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 1686688861, 21:41:01 DEBUG "0.00040000000000001514" 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG } 21:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688861" 21:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688861" 21:41:01 INFO [http_cmd]: http status code OK 21:41:01 INFO [http_cmd]: http status code OK 21:41:01 DEBUG --- status code --- 21:41:01 DEBUG --- status code --- 21:41:01 DEBUG http status code is 200 (expected 200) 21:41:01 DEBUG http status code is 200 (expected 200) 21:41:01 DEBUG --- http response --- 21:41:01 DEBUG --- http response --- 21:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:41:01 DEBUG ----- output ----- 21:41:01 DEBUG ----- output ----- 21:41:01 DEBUG { 21:41:01 DEBUG { 21:41:01 DEBUG "status": "success", 21:41:01 DEBUG "status": "success", 21:41:01 DEBUG "data": { 21:41:01 DEBUG "data": { 21:41:01 DEBUG "resultType": "vector", 21:41:01 DEBUG "resultType": "vector", 21:41:01 DEBUG "result": [ 21:41:01 DEBUG "result": [ 21:41:01 DEBUG { 21:41:01 DEBUG { 21:41:01 DEBUG "metric": { 21:41:01 DEBUG "metric": { 21:41:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:41:01 DEBUG "pod": "ds-idrepo-2" 21:41:01 DEBUG }, 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG [...] 21:41:01 DEBUG [...] 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:41:01 DEBUG "pod": "ds-idrepo-0" 21:41:01 DEBUG }, 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 1686688861, 21:41:01 DEBUG 1686688861, 21:41:01 DEBUG "0.2317132712807521" 21:41:01 DEBUG "606.1" 21:41:01 DEBUG ] 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG } 21:41:01 DEBUG ] 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG } 21:41:01 DEBUG } 21:41:01 DEBUG } 21:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688861" 21:41:01 INFO [http_cmd]: http status code OK 21:41:01 DEBUG --- status code --- 21:41:01 DEBUG http status code is 200 (expected 200) 21:41:01 DEBUG --- http response --- 21:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:41:01 DEBUG ----- output ----- 21:41:01 DEBUG { 21:41:01 DEBUG "status": "success", 21:41:01 DEBUG "data": { 21:41:01 DEBUG "resultType": "vector", 21:41:01 DEBUG "result": [ 21:41:01 DEBUG { 21:41:01 DEBUG "metric": { 21:41:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG [...] 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG 21:41:01 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 21:41:01 DEBUG }, 21:41:01 DEBUG "value": [ 21:41:01 DEBUG 1686688861, 21:41:01 DEBUG "0" 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG ] 21:41:01 DEBUG } 21:41:01 DEBUG } 21:41:15 INFO 21:41:15 INFO [loop_until]: kubectl --namespace=xlou top pods 21:41:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:41:15 INFO [loop_until]: OK (rc = 0) 21:41:15 DEBUG --- stdout --- 21:41:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 5m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3568m 13821Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 13m 12908Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3267m 4836Mi idm-58dc667486-rgz9j 2773m 5089Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 213m 637Mi 21:41:15 DEBUG --- stderr --- 21:41:15 DEBUG 21:41:18 INFO 21:41:18 INFO [loop_until]: kubectl --namespace=xlou top node 21:41:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:41:19 INFO [loop_until]: OK (rc = 0) 21:41:19 DEBUG --- stdout --- 21:41:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3415m 21% 5801Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6818Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2745m 17% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 450m 2% 3138Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1474Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13703Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3551m 22% 14500Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14581Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 279m 1% 1955Mi 3% 21:41:19 DEBUG --- stderr --- 21:41:19 DEBUG 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG { 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "data": { 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "result": [ 21:42:00 DEBUG { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "pod": "ds-cts-0" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG "pod": "ds-idrepo-0" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "643.3" 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG { 21:42:00 DEBUG { 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "data": { 21:42:00 DEBUG "data": { 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "result": [ 21:42:00 DEBUG "result": [ 21:42:00 DEBUG { 21:42:00 DEBUG { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "pod": "ds-cts-0" 21:42:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:42:00 DEBUG }, 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 DEBUG [...] 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG "pod": "ds-cts-2" 21:42:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:42:00 DEBUG }, 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "0" 21:42:00 DEBUG "0" 21:42:00 DEBUG ] 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 DEBUG ] 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG { 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "data": { 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "result": [ 21:42:00 DEBUG { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "0" 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG { 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "data": { 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "result": [ 21:42:00 DEBUG { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG [...] 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:42:00 DEBUG 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG 21:42:00 DEBUG { 21:42:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG }, 21:42:00 DEBUG "data": { 21:42:00 DEBUG "value": [ 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "result": [ 21:42:00 DEBUG "0.2" 21:42:00 DEBUG { 21:42:00 DEBUG ] 21:42:00 DEBUG "metric": { 21:42:00 DEBUG } 21:42:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:42:00 DEBUG ] 21:42:00 DEBUG }, 21:42:00 DEBUG } 21:42:00 DEBUG "value": [ 21:42:00 DEBUG } 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:42:00 DEBUG }, 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG "value": [ 21:42:00 DEBUG { 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "1.78694170808861" 21:42:00 DEBUG "data": { 21:42:00 DEBUG ] 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG } 21:42:00 DEBUG "result": [ 21:42:00 DEBUG ] 21:42:00 DEBUG { 21:42:00 DEBUG } 21:42:00 DEBUG "metric": { 21:42:00 DEBUG } 21:42:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "142724028.38912538" 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688920" 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG { 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "data": { 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "result": [ 21:42:00 DEBUG { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "instance": "10.142.0.100:9100" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG "instance": "10.142.0.99:9100" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "0" 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG { 21:42:00 DEBUG { 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "data": { 21:42:00 DEBUG "data": { 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "result": [ 21:42:00 DEBUG "result": [ 21:42:00 DEBUG { 21:42:00 DEBUG { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:42:00 DEBUG "pod": "ds-idrepo-2" 21:42:00 DEBUG }, 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 DEBUG [...] 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:42:00 DEBUG "pod": "ds-idrepo-0" 21:42:00 DEBUG }, 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "0" 21:42:00 DEBUG "643.1666666666666" 21:42:00 DEBUG ] 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 DEBUG ] 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG { 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "data": { 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "result": [ 21:42:00 DEBUG { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "pod": "ds-cts-0" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG "pod": "ds-idrepo-0" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "0" 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688920" 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG { 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "data": { 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "result": [ 21:42:00 DEBUG { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG "pod": "idm-58dc667486-rgz9j" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "0" 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688920" 21:42:00 INFO [http_cmd]: http status code OK 21:42:00 DEBUG --- status code --- 21:42:00 DEBUG http status code is 200 (expected 200) 21:42:00 DEBUG --- http response --- 21:42:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:42:00 DEBUG ----- output ----- 21:42:00 DEBUG { 21:42:00 DEBUG "status": "success", 21:42:00 DEBUG "data": { 21:42:00 DEBUG "resultType": "vector", 21:42:00 DEBUG "result": [ 21:42:00 DEBUG { 21:42:00 DEBUG "metric": { 21:42:00 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG [...] 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG 21:42:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:42:00 DEBUG }, 21:42:00 DEBUG "value": [ 21:42:00 DEBUG 1686688920, 21:42:00 DEBUG "13613674496" 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG ] 21:42:00 DEBUG } 21:42:00 DEBUG } 21:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688921" 21:42:01 INFO [http_cmd]: http status code OK 21:42:01 DEBUG --- status code --- 21:42:01 DEBUG http status code is 200 (expected 200) 21:42:01 DEBUG --- http response --- 21:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:42:01 DEBUG ----- output ----- 21:42:01 DEBUG { 21:42:01 DEBUG "status": "success", 21:42:01 DEBUG "data": { 21:42:01 DEBUG "resultType": "vector", 21:42:01 DEBUG "result": [ 21:42:01 DEBUG { 21:42:01 DEBUG "metric": { 21:42:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG [...] 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 1686688921, 21:42:01 DEBUG "0" 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG } 21:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688921" 21:42:01 INFO [http_cmd]: http status code OK 21:42:01 DEBUG --- status code --- 21:42:01 DEBUG http status code is 200 (expected 200) 21:42:01 DEBUG --- http response --- 21:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:42:01 DEBUG ----- output ----- 21:42:01 DEBUG { 21:42:01 DEBUG "status": "success", 21:42:01 DEBUG "data": { 21:42:01 DEBUG "resultType": "vector", 21:42:01 DEBUG "result": [ 21:42:01 DEBUG { 21:42:01 DEBUG "metric": { 21:42:01 DEBUG "pod": "ds-idrepo-2" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG [...] 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG "pod": "ds-idrepo-0" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 1686688921, 21:42:01 DEBUG "643.1333333333333" 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG } 21:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688921" 21:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688921" 21:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688921" 21:42:01 INFO [http_cmd]: http status code OK 21:42:01 DEBUG --- status code --- 21:42:01 DEBUG http status code is 200 (expected 200) 21:42:01 DEBUG --- http response --- 21:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:42:01 DEBUG ----- output ----- 21:42:01 DEBUG { 21:42:01 DEBUG "status": "success", 21:42:01 DEBUG "data": { 21:42:01 DEBUG "resultType": "vector", 21:42:01 DEBUG "result": [ 21:42:01 DEBUG { 21:42:01 DEBUG "metric": { 21:42:01 DEBUG "instance": "10.142.0.100:9100" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG [...] 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG "instance": "10.142.0.99:9100" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 1686688921, 21:42:01 DEBUG "0.00043333333333445503" 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG } 21:42:01 INFO [http_cmd]: http status code OK 21:42:01 DEBUG --- status code --- 21:42:01 DEBUG http status code is 200 (expected 200) 21:42:01 INFO [http_cmd]: http status code OK 21:42:01 DEBUG --- http response --- 21:42:01 DEBUG --- status code --- 21:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:42:01 DEBUG http status code is 200 (expected 200) 21:42:01 DEBUG ----- output ----- 21:42:01 DEBUG --- http response --- 21:42:01 DEBUG { 21:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:42:01 DEBUG "status": "success", 21:42:01 DEBUG ----- output ----- 21:42:01 DEBUG "data": { 21:42:01 DEBUG { 21:42:01 DEBUG "resultType": "vector", 21:42:01 DEBUG "status": "success", 21:42:01 DEBUG "result": [ 21:42:01 DEBUG "data": { 21:42:01 DEBUG { 21:42:01 DEBUG "resultType": "vector", 21:42:01 DEBUG "metric": { 21:42:01 DEBUG "result": [ 21:42:01 DEBUG "instance": "10.142.0.100:9100" 21:42:01 DEBUG { 21:42:01 DEBUG }, 21:42:01 DEBUG "metric": { 21:42:01 DEBUG "value": [ 21:42:01 DEBUG "instance": "10.142.0.100:9100" 21:42:01 DEBUG 21:42:01 DEBUG }, 21:42:01 DEBUG 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG [...] 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG [...] 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG "instance": "10.142.0.99:9100" 21:42:01 DEBUG 21:42:01 DEBUG }, 21:42:01 DEBUG 21:42:01 DEBUG "value": [ 21:42:01 DEBUG "instance": "10.142.0.99:9100" 21:42:01 DEBUG 1686688921, 21:42:01 DEBUG }, 21:42:01 DEBUG "20175196.9" 21:42:01 DEBUG "value": [ 21:42:01 DEBUG ] 21:42:01 DEBUG 1686688921, 21:42:01 DEBUG } 21:42:01 DEBUG "20176998.433333334" 21:42:01 DEBUG ] 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG } 21:42:01 DEBUG } 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG } 21:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688921" 21:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688921" 21:42:01 INFO [http_cmd]: http status code OK 21:42:01 DEBUG --- status code --- 21:42:01 DEBUG http status code is 200 (expected 200) 21:42:01 DEBUG --- http response --- 21:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:42:01 DEBUG ----- output ----- 21:42:01 DEBUG { 21:42:01 DEBUG "status": "success", 21:42:01 DEBUG "data": { 21:42:01 DEBUG "resultType": "vector", 21:42:01 DEBUG "result": [ 21:42:01 DEBUG { 21:42:01 DEBUG "metric": { 21:42:01 DEBUG "pod": "ds-idrepo-2" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG [...] 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 INFO [http_cmd]: http status code OK 21:42:01 DEBUG "pod": "ds-idrepo-0" 21:42:01 DEBUG --- status code --- 21:42:01 DEBUG }, 21:42:01 DEBUG http status code is 200 (expected 200) 21:42:01 DEBUG "value": [ 21:42:01 DEBUG --- http response --- 21:42:01 DEBUG 1686688921, 21:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:42:01 DEBUG "643.1333333333333" 21:42:01 DEBUG ----- output ----- 21:42:01 DEBUG ] 21:42:01 DEBUG { 21:42:01 DEBUG } 21:42:01 DEBUG "status": "success", 21:42:01 DEBUG ] 21:42:01 DEBUG "data": { 21:42:01 DEBUG } 21:42:01 DEBUG "resultType": "vector", 21:42:01 DEBUG } 21:42:01 DEBUG "result": [ 21:42:01 DEBUG { 21:42:01 DEBUG "metric": { 21:42:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG [...] 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 1686688921, 21:42:01 DEBUG "0.23447603743390458" 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG } 21:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688921" 21:42:01 INFO [http_cmd]: http status code OK 21:42:01 DEBUG --- status code --- 21:42:01 DEBUG http status code is 200 (expected 200) 21:42:01 DEBUG --- http response --- 21:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:42:01 DEBUG ----- output ----- 21:42:01 DEBUG { 21:42:01 DEBUG "status": "success", 21:42:01 DEBUG "data": { 21:42:01 DEBUG "resultType": "vector", 21:42:01 DEBUG "result": [ 21:42:01 DEBUG { 21:42:01 DEBUG "metric": { 21:42:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG [...] 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG 21:42:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:42:01 DEBUG }, 21:42:01 DEBUG "value": [ 21:42:01 DEBUG 1686688921, 21:42:01 DEBUG "0" 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG ] 21:42:01 DEBUG } 21:42:01 DEBUG } 21:42:15 INFO 21:42:15 INFO [loop_until]: kubectl --namespace=xlou top pods 21:42:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:42:15 INFO [loop_until]: OK (rc = 0) 21:42:15 DEBUG --- stdout --- 21:42:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3354m 13803Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3075m 4839Mi idm-58dc667486-rgz9j 2793m 5089Mi lodemon-6455d5ff7c-zzndq 5m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 198m 638Mi 21:42:15 DEBUG --- stderr --- 21:42:15 DEBUG 21:42:19 INFO 21:42:19 INFO [loop_until]: kubectl --namespace=xlou top node 21:42:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:42:19 INFO [loop_until]: OK (rc = 0) 21:42:19 DEBUG --- stdout --- 21:42:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3222m 20% 5791Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 68m 0% 6817Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2809m 17% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 69m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 449m 2% 3146Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 85m 0% 1475Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1163Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1191Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13705Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3482m 21% 14498Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1163Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14583Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 272m 1% 1953Mi 3% 21:42:19 DEBUG --- stderr --- 21:42:19 DEBUG 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG { 21:43:00 DEBUG { 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "data": { 21:43:00 DEBUG "data": { 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "result": [ 21:43:00 DEBUG "result": [ 21:43:00 DEBUG { 21:43:00 DEBUG { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "pod": "ds-cts-0" 21:43:00 DEBUG "pod": "ds-cts-0" 21:43:00 DEBUG }, 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG [...] 21:43:00 DEBUG [...] 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG "pod": "ds-idrepo-0" 21:43:00 DEBUG "pod": "ds-cts-2" 21:43:00 DEBUG }, 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG "606.8666666666667" 21:43:00 DEBUG "0" 21:43:00 DEBUG ] 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 DEBUG ] 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG { 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "data": { 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "result": [ 21:43:00 DEBUG { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG [...] 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG "0" 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG { 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "data": { 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "result": [ 21:43:00 DEBUG { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "pod": "ds-cts-2" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG [...] 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG "pod": "am-7849cf7bdb-lzlql" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG "0" 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG { 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "data": { 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "result": [ 21:43:00 DEBUG { 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "pod": "ds-cts-2" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG [...] 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG "pod": "am-7849cf7bdb-lzlql" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG "0" 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG { 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "data": { 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "result": [ 21:43:00 DEBUG { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG [...] 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG "0" 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG { 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "data": { 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "result": [ 21:43:00 DEBUG { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG [...] 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG "0.2" 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688980" 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG { 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "data": { 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "result": [ 21:43:00 DEBUG { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "instance": "10.142.0.100:9100" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG [...] 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG "instance": "10.142.0.99:9100" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG "0" 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG { 21:43:00 DEBUG { 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "data": { 21:43:00 DEBUG "data": { 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "result": [ 21:43:00 DEBUG "result": [ 21:43:00 DEBUG { 21:43:00 DEBUG { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:43:00 DEBUG "pod": "ds-idrepo-2" 21:43:00 DEBUG }, 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG [...] 21:43:00 DEBUG [...] 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:43:00 DEBUG "pod": "ds-idrepo-0" 21:43:00 DEBUG }, 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG "0" 21:43:00 DEBUG "606.8333333333334" 21:43:00 DEBUG ] 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 DEBUG ] 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:00 INFO [http_cmd]: http status code OK 21:43:00 DEBUG --- status code --- 21:43:00 DEBUG http status code is 200 (expected 200) 21:43:00 DEBUG --- http response --- 21:43:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:43:00 DEBUG ----- output ----- 21:43:00 DEBUG { 21:43:00 DEBUG "status": "success", 21:43:00 DEBUG "data": { 21:43:00 DEBUG "resultType": "vector", 21:43:00 DEBUG "result": [ 21:43:00 DEBUG { 21:43:00 DEBUG "metric": { 21:43:00 DEBUG "pod": "ds-cts-0" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG [...] 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG 21:43:00 DEBUG "pod": "ds-idrepo-0" 21:43:00 DEBUG }, 21:43:00 DEBUG "value": [ 21:43:00 DEBUG 1686688980, 21:43:00 DEBUG "0" 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG ] 21:43:00 DEBUG } 21:43:00 DEBUG } 21:43:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688980" 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "pod": "ds-cts-2" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688980, 21:43:01 DEBUG "0" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686688981" 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688981, 21:43:01 DEBUG "13616619520" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688981" 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688981, 21:43:01 DEBUG "0" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688981" 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "pod": "ds-idrepo-2" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "pod": "ds-idrepo-0" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688981, 21:43:01 DEBUG "606.8333333333334" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688981" 21:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688981" 21:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686688981" 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "instance": "10.142.0.100:9100" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "instance": "10.142.0.99:9100" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688981, 21:43:01 DEBUG "0.0004333333333325603" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "instance": "10.142.0.100:9100" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "instance": "10.142.0.99:9100" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688981, 21:43:01 DEBUG "22362672.799999997" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "instance": "10.142.0.100:9100" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "instance": "10.142.0.99:9100" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688981, 21:43:01 DEBUG "22362828.133333333" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686688981" 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688981, 21:43:01 DEBUG "0.2372134506416682" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688981" 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "pod": "ds-idrepo-2" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "pod": "ds-idrepo-0" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688981, 21:43:01 DEBUG "606.8333333333334" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686688981" 21:43:01 INFO [http_cmd]: http status code OK 21:43:01 DEBUG --- status code --- 21:43:01 DEBUG http status code is 200 (expected 200) 21:43:01 DEBUG --- http response --- 21:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:43:01 DEBUG ----- output ----- 21:43:01 DEBUG { 21:43:01 DEBUG "status": "success", 21:43:01 DEBUG "data": { 21:43:01 DEBUG "resultType": "vector", 21:43:01 DEBUG "result": [ 21:43:01 DEBUG { 21:43:01 DEBUG "metric": { 21:43:01 DEBUG "pod": "ds-cts-2" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG [...] 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG 21:43:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:43:01 DEBUG }, 21:43:01 DEBUG "value": [ 21:43:01 DEBUG 1686688981, 21:43:01 DEBUG "0" 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG ] 21:43:01 DEBUG } 21:43:01 DEBUG } 21:43:16 INFO 21:43:16 INFO [loop_until]: kubectl --namespace=xlou top pods 21:43:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:43:16 INFO [loop_until]: OK (rc = 0) 21:43:16 DEBUG --- stdout --- 21:43:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 10m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3192m 13798Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2693m 4840Mi idm-58dc667486-rgz9j 2798m 5089Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 200m 637Mi 21:43:16 DEBUG --- stderr --- 21:43:16 DEBUG 21:43:19 INFO 21:43:19 INFO [loop_until]: kubectl --namespace=xlou top node 21:43:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:43:19 INFO [loop_until]: OK (rc = 0) 21:43:19 DEBUG --- stdout --- 21:43:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2821m 17% 5787Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6817Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2911m 18% 6083Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 440m 2% 3145Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 86m 0% 1475Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 59m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1194Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 66m 0% 13706Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3209m 20% 14492Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 67m 0% 1171Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14587Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 265m 1% 1948Mi 3% 21:43:19 DEBUG --- stderr --- 21:43:19 DEBUG 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG { 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG "data": { 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "result": [ 21:44:00 DEBUG { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "pod": "ds-cts-0" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG [...] 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG "pod": "ds-idrepo-0" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG "604.8333333333334" 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG { 21:44:00 DEBUG { 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG "data": { 21:44:00 DEBUG "data": { 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "result": [ 21:44:00 DEBUG "result": [ 21:44:00 DEBUG { 21:44:00 DEBUG { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "pod": "ds-cts-0" 21:44:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:44:00 DEBUG }, 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG [...] 21:44:00 DEBUG [...] 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG "pod": "ds-cts-2" 21:44:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:44:00 DEBUG }, 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG "0" 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG "0" 21:44:00 DEBUG ] 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG { 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG } 21:44:00 DEBUG "data": { 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "result": [ 21:44:00 DEBUG { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG [...] 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG "0.0883625978243671" 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG --- http response --- 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG { 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:44:00 DEBUG "data": { 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG { 21:44:00 DEBUG "result": [ 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG { 21:44:00 DEBUG "data": { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:44:00 DEBUG "result": [ 21:44:00 DEBUG }, 21:44:00 DEBUG { 21:44:00 DEBUG "value": [ 21:44:00 DEBUG "metric": { 21:44:00 DEBUG 21:44:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:44:00 DEBUG 21:44:00 DEBUG }, 21:44:00 DEBUG 21:44:00 DEBUG "value": [ 21:44:00 DEBUG [...] 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG [...] 21:44:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:44:00 DEBUG 21:44:00 DEBUG }, 21:44:00 DEBUG 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 21:44:00 DEBUG "0" 21:44:00 DEBUG }, 21:44:00 DEBUG ] 21:44:00 DEBUG "value": [ 21:44:00 DEBUG } 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG ] 21:44:00 DEBUG "3950.8733775472238" 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG { 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG "data": { 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "result": [ 21:44:00 DEBUG { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG [...] 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG "0.2" 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG { 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG "data": { 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "result": [ 21:44:00 DEBUG { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "pod": "ds-idrepo-2" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG [...] 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG "pod": "ds-idrepo-0" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG "604.8666666666667" 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689040" 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG { 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG "data": { 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "result": [ 21:44:00 DEBUG { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "instance": "10.142.0.100:9100" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG [...] 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG "instance": "10.142.0.99:9100" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG "0" 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG { 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG "data": { 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "result": [ 21:44:00 DEBUG { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "pod": "ds-cts-0" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG [...] 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG "pod": "ds-idrepo-0" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG "0" 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689040" 21:44:00 INFO [http_cmd]: http status code OK 21:44:00 DEBUG --- status code --- 21:44:00 DEBUG http status code is 200 (expected 200) 21:44:00 DEBUG --- http response --- 21:44:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:44:00 DEBUG ----- output ----- 21:44:00 DEBUG { 21:44:00 DEBUG "status": "success", 21:44:00 DEBUG "data": { 21:44:00 DEBUG "resultType": "vector", 21:44:00 DEBUG "result": [ 21:44:00 DEBUG { 21:44:00 DEBUG "metric": { 21:44:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG [...] 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG 21:44:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:44:00 DEBUG }, 21:44:00 DEBUG "value": [ 21:44:00 DEBUG 1686689040, 21:44:00 DEBUG "0" 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG ] 21:44:00 DEBUG } 21:44:00 DEBUG } 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689041" 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "13616201728" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689041" 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "pod": "ds-idrepo-2" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "0" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689041" 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "0" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689041" 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "pod": "ds-idrepo-2" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "pod": "ds-idrepo-0" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "604.8666666666667" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689041" 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "instance": "10.142.0.100:9100" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "instance": "10.142.0.99:9100" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "20319701.066666666" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689041" 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689041" 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "instance": "10.142.0.100:9100" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "instance": "10.142.0.99:9100" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "0.00040000000000001514" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "instance": "10.142.0.100:9100" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "instance": "10.142.0.99:9100" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "20317476.566666666" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689041" 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "0.23027575654742477" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689041" 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "pod": "ds-idrepo-2" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "pod": "ds-idrepo-0" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "604.8333333333334" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689041" 21:44:01 INFO [http_cmd]: http status code OK 21:44:01 DEBUG --- status code --- 21:44:01 DEBUG http status code is 200 (expected 200) 21:44:01 DEBUG --- http response --- 21:44:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:44:01 DEBUG ----- output ----- 21:44:01 DEBUG { 21:44:01 DEBUG "status": "success", 21:44:01 DEBUG "data": { 21:44:01 DEBUG "resultType": "vector", 21:44:01 DEBUG "result": [ 21:44:01 DEBUG { 21:44:01 DEBUG "metric": { 21:44:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG [...] 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG 21:44:01 DEBUG "pod": "ds-idrepo-2" 21:44:01 DEBUG }, 21:44:01 DEBUG "value": [ 21:44:01 DEBUG 1686689041, 21:44:01 DEBUG "0.6810700367021075" 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG ] 21:44:01 DEBUG } 21:44:01 DEBUG } 21:44:16 INFO 21:44:16 INFO [loop_until]: kubectl --namespace=xlou top pods 21:44:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:44:16 INFO [loop_until]: OK (rc = 0) 21:44:16 DEBUG --- stdout --- 21:44:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3492m 13805Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2738m 4842Mi idm-58dc667486-rgz9j 2690m 5090Mi lodemon-6455d5ff7c-zzndq 4m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 204m 638Mi 21:44:16 DEBUG --- stderr --- 21:44:16 DEBUG 21:44:19 INFO 21:44:19 INFO [loop_until]: kubectl --namespace=xlou top node 21:44:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:44:19 INFO [loop_until]: OK (rc = 0) 21:44:19 DEBUG --- stdout --- 21:44:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2933m 18% 5790Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2915m 18% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 445m 2% 3147Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1476Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1196Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13708Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3388m 21% 14499Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14585Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 275m 1% 1955Mi 3% 21:44:19 DEBUG --- stderr --- 21:44:19 DEBUG 21:45:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689100" 21:45:00 INFO [http_cmd]: http status code OK 21:45:00 DEBUG --- status code --- 21:45:00 DEBUG http status code is 200 (expected 200) 21:45:00 DEBUG --- http response --- 21:45:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:45:00 DEBUG ----- output ----- 21:45:00 DEBUG { 21:45:00 DEBUG "status": "success", 21:45:00 DEBUG "data": { 21:45:00 DEBUG "resultType": "vector", 21:45:00 DEBUG "result": [ 21:45:00 DEBUG { 21:45:00 DEBUG "metric": { 21:45:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG [...] 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 1686689100, 21:45:00 DEBUG "0.005543966001971411" 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG } 21:45:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689100" 21:45:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689100" 21:45:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689100" 21:45:00 INFO [http_cmd]: http status code OK 21:45:00 DEBUG --- status code --- 21:45:00 DEBUG http status code is 200 (expected 200) 21:45:00 DEBUG --- http response --- 21:45:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:45:00 DEBUG ----- output ----- 21:45:00 DEBUG { 21:45:00 DEBUG "status": "success", 21:45:00 DEBUG "data": { 21:45:00 DEBUG "resultType": "vector", 21:45:00 DEBUG "result": [ 21:45:00 DEBUG { 21:45:00 DEBUG "metric": { 21:45:00 DEBUG "pod": "ds-cts-0" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 21:45:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689100" 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG [...] 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG "pod": "ds-idrepo-0" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 1686689100, 21:45:00 DEBUG "601.3333333333334" 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG } 21:45:00 INFO [http_cmd]: http status code OK 21:45:00 DEBUG --- status code --- 21:45:00 DEBUG http status code is 200 (expected 200) 21:45:00 DEBUG --- http response --- 21:45:00 INFO [http_cmd]: http status code OK 21:45:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:45:00 DEBUG --- status code --- 21:45:00 DEBUG ----- output ----- 21:45:00 DEBUG http status code is 200 (expected 200) 21:45:00 DEBUG { 21:45:00 DEBUG --- http response --- 21:45:00 DEBUG "status": "success", 21:45:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:45:00 DEBUG "data": { 21:45:00 DEBUG ----- output ----- 21:45:00 DEBUG "resultType": "vector", 21:45:00 DEBUG { 21:45:00 DEBUG "result": [ 21:45:00 DEBUG "status": "success", 21:45:00 DEBUG { 21:45:00 DEBUG "data": { 21:45:00 DEBUG "metric": { 21:45:00 DEBUG "resultType": "vector", 21:45:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:45:00 DEBUG "result": [ 21:45:00 DEBUG }, 21:45:00 DEBUG { 21:45:00 DEBUG "value": [ 21:45:00 DEBUG "metric": { 21:45:00 DEBUG 21:45:00 DEBUG "pod": "ds-cts-0" 21:45:00 DEBUG 21:45:00 DEBUG }, 21:45:00 DEBUG 21:45:00 DEBUG "value": [ 21:45:00 DEBUG [...] 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG [...] 21:45:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:45:00 DEBUG 21:45:00 DEBUG }, 21:45:00 DEBUG 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 21:45:00 DEBUG 1686689100, 21:45:00 DEBUG "pod": "ds-cts-2" 21:45:00 DEBUG "0" 21:45:00 DEBUG }, 21:45:00 DEBUG ] 21:45:00 DEBUG "value": [ 21:45:00 DEBUG } 21:45:00 DEBUG 1686689100, 21:45:00 DEBUG ] 21:45:00 DEBUG "0" 21:45:00 DEBUG } 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG } 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG } 21:45:00 INFO [http_cmd]: http status code OK 21:45:00 DEBUG --- status code --- 21:45:00 DEBUG http status code is 200 (expected 200) 21:45:00 DEBUG --- http response --- 21:45:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:45:00 DEBUG ----- output ----- 21:45:00 DEBUG { 21:45:00 DEBUG "status": "success", 21:45:00 DEBUG "data": { 21:45:00 DEBUG "resultType": "vector", 21:45:00 DEBUG "result": [ 21:45:00 DEBUG { 21:45:00 DEBUG "metric": { 21:45:00 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG [...] 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 1686689100, 21:45:00 DEBUG "189734046.24240687" 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG } 21:45:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689100" 21:45:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689100" 21:45:00 INFO [http_cmd]: http status code OK 21:45:00 DEBUG --- status code --- 21:45:00 DEBUG http status code is 200 (expected 200) 21:45:00 DEBUG --- http response --- 21:45:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:45:00 DEBUG ----- output ----- 21:45:00 DEBUG { 21:45:00 DEBUG "status": "success", 21:45:00 DEBUG "data": { 21:45:00 DEBUG "resultType": "vector", 21:45:00 DEBUG "result": [ 21:45:00 DEBUG { 21:45:00 DEBUG "metric": { 21:45:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG [...] 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 1686689100, 21:45:00 DEBUG "0" 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG } 21:45:00 INFO [http_cmd]: http status code OK 21:45:00 DEBUG --- status code --- 21:45:00 DEBUG http status code is 200 (expected 200) 21:45:00 DEBUG --- http response --- 21:45:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:45:00 DEBUG ----- output ----- 21:45:00 DEBUG { 21:45:00 DEBUG "status": "success", 21:45:00 DEBUG "data": { 21:45:00 DEBUG "resultType": "vector", 21:45:00 DEBUG "result": [ 21:45:00 DEBUG { 21:45:00 DEBUG "metric": { 21:45:00 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG [...] 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG 21:45:00 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:45:00 DEBUG }, 21:45:00 DEBUG "value": [ 21:45:00 DEBUG 1686689100, 21:45:00 DEBUG "0.2" 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG ] 21:45:00 DEBUG } 21:45:00 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "pod": "ds-idrepo-2" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "pod": "ds-idrepo-0" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "601.3" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689101" 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "instance": "10.142.0.100:9100" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "instance": "10.142.0.99:9100" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "0" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "pod": "ds-cts-0" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "pod": "ds-idrepo-0" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "0" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "0" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "13616930816" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "pod": "ds-cts-2" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "0" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "0" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "pod": "ds-idrepo-2" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "pod": "ds-idrepo-0" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "601.3" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "instance": "10.142.0.100:9100" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "instance": "10.142.0.99:9100" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "19385822.833333332" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "instance": "10.142.0.100:9100" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "instance": "10.142.0.99:9100" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "19383317.933333334" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "instance": "10.142.0.100:9100" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "instance": "10.142.0.99:9100" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "0.00023333333333350008" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "pod": "ds-idrepo-2" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "pod": "ds-idrepo-0" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "601.3" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "0.21974907043569514" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689101" 21:45:01 INFO [http_cmd]: http status code OK 21:45:01 DEBUG --- status code --- 21:45:01 DEBUG http status code is 200 (expected 200) 21:45:01 DEBUG --- http response --- 21:45:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:45:01 DEBUG ----- output ----- 21:45:01 DEBUG { 21:45:01 DEBUG "status": "success", 21:45:01 DEBUG "data": { 21:45:01 DEBUG "resultType": "vector", 21:45:01 DEBUG "result": [ 21:45:01 DEBUG { 21:45:01 DEBUG "metric": { 21:45:01 DEBUG "pod": "ds-cts-2" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG [...] 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG 21:45:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:45:01 DEBUG }, 21:45:01 DEBUG "value": [ 21:45:01 DEBUG 1686689101, 21:45:01 DEBUG "0" 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG ] 21:45:01 DEBUG } 21:45:01 DEBUG } 21:45:16 INFO 21:45:16 INFO [loop_until]: kubectl --namespace=xlou top pods 21:45:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:45:16 INFO [loop_until]: OK (rc = 0) 21:45:16 DEBUG --- stdout --- 21:45:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 10m 416Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3343m 13797Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3058m 4845Mi idm-58dc667486-rgz9j 2503m 5090Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 195m 638Mi 21:45:16 DEBUG --- stderr --- 21:45:16 DEBUG 21:45:19 INFO 21:45:19 INFO [loop_until]: kubectl --namespace=xlou top node 21:45:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:45:19 INFO [loop_until]: OK (rc = 0) 21:45:19 DEBUG --- stdout --- 21:45:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3133m 19% 5795Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6852Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6813Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2571m 16% 6075Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 68m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 435m 2% 3149Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 85m 0% 1474Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1171Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13708Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3229m 20% 14497Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14583Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 263m 1% 1956Mi 3% 21:45:19 DEBUG --- stderr --- 21:45:19 DEBUG 21:46:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689160" 21:46:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689160" 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:46:01 DEBUG { 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG { 21:46:01 DEBUG "data": { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "result": [ 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG { 21:46:01 DEBUG "result": [ 21:46:01 DEBUG "metric": { 21:46:01 DEBUG { 21:46:01 DEBUG "pod": "ds-cts-0" 21:46:01 DEBUG "metric": { 21:46:01 DEBUG }, 21:46:01 DEBUG "pod": "ds-cts-2" 21:46:01 DEBUG "value": [ 21:46:01 DEBUG }, 21:46:01 DEBUG 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "ds-idrepo-0" 21:46:01 DEBUG 21:46:01 DEBUG }, 21:46:01 DEBUG "pod": "am-7849cf7bdb-lzlql" 21:46:01 DEBUG "value": [ 21:46:01 DEBUG }, 21:46:01 DEBUG 1686689160, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG "565.6333333333333" 21:46:01 DEBUG 1686689160, 21:46:01 DEBUG ] 21:46:01 DEBUG "0" 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:46:01 DEBUG { 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG { 21:46:01 DEBUG { 21:46:01 DEBUG "data": { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "data": { 21:46:01 DEBUG "result": [ 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG { 21:46:01 DEBUG "result": [ 21:46:01 DEBUG "result": [ 21:46:01 DEBUG "metric": { 21:46:01 DEBUG { 21:46:01 DEBUG { 21:46:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG }, 21:46:01 DEBUG "pod": "ds-cts-0" 21:46:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:46:01 DEBUG "value": [ 21:46:01 DEBUG }, 21:46:01 DEBUG }, 21:46:01 DEBUG 21:46:01 DEBUG "value": [ 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG { 21:46:01 DEBUG 21:46:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:46:01 DEBUG 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG 21:46:01 DEBUG }, 21:46:01 DEBUG "pod": "ds-cts-2" 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG "data": { 21:46:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:46:01 DEBUG "value": [ 21:46:01 DEBUG }, 21:46:01 DEBUG { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG }, 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG "value": [ 21:46:01 DEBUG "0.2" 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "data": { 21:46:01 DEBUG { 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG ] 21:46:01 DEBUG "0" 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "0" 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG "result": [ 21:46:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:46:01 DEBUG ] 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG { 21:46:01 DEBUG }, 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "value": [ 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 DEBUG "pod": "ds-cts-2" 21:46:01 DEBUG 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 DEBUG }, 21:46:01 DEBUG 21:46:01 DEBUG } 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:46:01 DEBUG 21:46:01 DEBUG }, 21:46:01 DEBUG 21:46:01 DEBUG "value": [ 21:46:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG }, 21:46:01 DEBUG "0" 21:46:01 DEBUG "value": [ 21:46:01 DEBUG ] 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG } 21:46:01 DEBUG "0" 21:46:01 DEBUG ] 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "0" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "pod": "ds-idrepo-2" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "ds-idrepo-0" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "565.6666666666666" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689161" 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "pod": "ds-cts-0" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "ds-idrepo-0" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "0" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "instance": "10.142.0.100:9100" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "instance": "10.142.0.99:9100" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "0" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "13614870528" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "pod": "ds-cts-2" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "0" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "0" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "pod": "ds-idrepo-2" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "ds-idrepo-0" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "565.6666666666666" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "instance": "10.142.0.100:9100" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "instance": "10.142.0.99:9100" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "19315573.666666668" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "instance": "10.142.0.100:9100" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "instance": "10.142.0.99:9100" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "19315321" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "instance": "10.142.0.100:9100" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "instance": "10.142.0.99:9100" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "0.0004333333333325603" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "pod": "ds-idrepo-2" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "ds-idrepo-0" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "565.6666666666666" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "0.22766706034657047" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689161" 21:46:01 INFO [http_cmd]: http status code OK 21:46:01 DEBUG --- status code --- 21:46:01 DEBUG http status code is 200 (expected 200) 21:46:01 DEBUG --- http response --- 21:46:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:46:01 DEBUG ----- output ----- 21:46:01 DEBUG { 21:46:01 DEBUG "status": "success", 21:46:01 DEBUG "data": { 21:46:01 DEBUG "resultType": "vector", 21:46:01 DEBUG "result": [ 21:46:01 DEBUG { 21:46:01 DEBUG "metric": { 21:46:01 DEBUG "pod": "ds-cts-2" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG [...] 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG 21:46:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:46:01 DEBUG }, 21:46:01 DEBUG "value": [ 21:46:01 DEBUG 1686689161, 21:46:01 DEBUG "0" 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG ] 21:46:01 DEBUG } 21:46:01 DEBUG } 21:46:16 INFO 21:46:16 INFO [loop_until]: kubectl --namespace=xlou top pods 21:46:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:46:16 INFO [loop_until]: OK (rc = 0) 21:46:16 DEBUG --- stdout --- 21:46:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3117m 13798Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2849m 4846Mi idm-58dc667486-rgz9j 2541m 5090Mi lodemon-6455d5ff7c-zzndq 2m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 192m 637Mi 21:46:16 DEBUG --- stderr --- 21:46:16 DEBUG 21:46:19 INFO 21:46:19 INFO [loop_until]: kubectl --namespace=xlou top node 21:46:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:46:19 INFO [loop_until]: OK (rc = 0) 21:46:19 DEBUG --- stdout --- 21:46:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2846m 17% 5798Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 67m 0% 6815Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2689m 16% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 427m 2% 3148Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 85m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1196Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13705Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3261m 20% 14501Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14582Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 257m 1% 1953Mi 3% 21:46:19 DEBUG --- stderr --- 21:46:19 DEBUG 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "ds-cts-0" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "ds-idrepo-0" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "615.8" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0.2" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "ds-cts-0" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "ds-cts-2" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "ds-cts-2" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0.00001665614089242776" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "ds-cts-2" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:47:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:47:01 DEBUG }, 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:47:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:47:01 DEBUG }, 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0" 21:47:01 DEBUG "0" 21:47:01 DEBUG ] 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "ds-cts-0" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "ds-idrepo-0" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "ds-idrepo-2" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "ds-idrepo-0" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "615.8" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "instance": "10.142.0.100:9100" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "instance": "10.142.0.99:9100" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "13615808512" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "ds-cts-2" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "ds-idrepo-2" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "ds-idrepo-0" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "615.7666666666667" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "instance": "10.142.0.100:9100" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "instance": "10.142.0.99:9100" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "20354923.4" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "instance": "10.142.0.100:9100" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "instance": "10.142.0.99:9100" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "20356959.2" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "instance": "10.142.0.100:9100" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "instance": "10.142.0.99:9100" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0.00036666666666557526" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "pod": "ds-idrepo-2" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "pod": "ds-idrepo-0" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "615.7666666666667" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689221" 21:47:01 INFO [http_cmd]: http status code OK 21:47:01 DEBUG --- status code --- 21:47:01 DEBUG http status code is 200 (expected 200) 21:47:01 DEBUG --- http response --- 21:47:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:47:01 DEBUG ----- output ----- 21:47:01 DEBUG { 21:47:01 DEBUG "status": "success", 21:47:01 DEBUG "data": { 21:47:01 DEBUG "resultType": "vector", 21:47:01 DEBUG "result": [ 21:47:01 DEBUG { 21:47:01 DEBUG "metric": { 21:47:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG [...] 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG 21:47:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:47:01 DEBUG }, 21:47:01 DEBUG "value": [ 21:47:01 DEBUG 1686689221, 21:47:01 DEBUG "0.22152624734217266" 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG ] 21:47:01 DEBUG } 21:47:01 DEBUG } 21:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689222" 21:47:02 INFO [http_cmd]: http status code OK 21:47:02 DEBUG --- status code --- 21:47:02 DEBUG http status code is 200 (expected 200) 21:47:02 DEBUG --- http response --- 21:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:47:02 DEBUG ----- output ----- 21:47:02 DEBUG { 21:47:02 DEBUG "status": "success", 21:47:02 DEBUG "data": { 21:47:02 DEBUG "resultType": "vector", 21:47:02 DEBUG "result": [ 21:47:02 DEBUG { 21:47:02 DEBUG "metric": { 21:47:02 DEBUG "pod": "ds-cts-2" 21:47:02 DEBUG }, 21:47:02 DEBUG "value": [ 21:47:02 DEBUG 21:47:02 DEBUG 21:47:02 DEBUG 21:47:02 DEBUG [...] 21:47:02 DEBUG 21:47:02 DEBUG 21:47:02 DEBUG 21:47:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:47:02 DEBUG }, 21:47:02 DEBUG "value": [ 21:47:02 DEBUG 1686689222, 21:47:02 DEBUG "0" 21:47:02 DEBUG ] 21:47:02 DEBUG } 21:47:02 DEBUG ] 21:47:02 DEBUG } 21:47:02 DEBUG } 21:47:16 INFO 21:47:16 INFO [loop_until]: kubectl --namespace=xlou top pods 21:47:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:47:16 INFO [loop_until]: OK (rc = 0) 21:47:16 DEBUG --- stdout --- 21:47:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 11m 416Mi ds-cts-1 9m 423Mi ds-cts-2 8m 392Mi ds-idrepo-0 3299m 13822Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3185m 4848Mi idm-58dc667486-rgz9j 2580m 5090Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 200m 638Mi 21:47:16 DEBUG --- stderr --- 21:47:16 DEBUG 21:47:19 INFO 21:47:19 INFO [loop_until]: kubectl --namespace=xlou top node 21:47:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:47:19 INFO [loop_until]: OK (rc = 0) 21:47:19 DEBUG --- stdout --- 21:47:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3200m 20% 5798Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6852Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6816Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2612m 16% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 438m 2% 3153Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 59m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 65m 0% 13708Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3473m 21% 14498Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 66m 0% 1171Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 75m 0% 14587Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 267m 1% 1951Mi 3% 21:47:19 DEBUG --- stderr --- 21:47:19 DEBUG 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG { 21:48:01 DEBUG { 21:48:01 DEBUG { 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "data": { 21:48:01 DEBUG "data": { 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG "result": [ 21:48:01 DEBUG "result": [ 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG { 21:48:01 DEBUG { 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "metric": { 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:48:01 DEBUG "pod": "ds-cts-0" 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:48:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:48:01 DEBUG }, 21:48:01 DEBUG }, 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG }, 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG "value": [ 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG "value": [ 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG { 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG [...] 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG [...] 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "data": { 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "result": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:48:01 DEBUG "pod": "ds-idrepo-0" 21:48:01 DEBUG { 21:48:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:48:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:48:01 DEBUG }, 21:48:01 DEBUG }, 21:48:01 DEBUG "metric": { 21:48:01 DEBUG }, 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG "value": [ 21:48:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:48:01 DEBUG "value": [ 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG }, 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0" 21:48:01 DEBUG "671.5333333333333" 21:48:01 DEBUG "value": [ 21:48:01 DEBUG "0.2" 21:48:01 DEBUG "0" 21:48:01 DEBUG ] 21:48:01 DEBUG ] 21:48:01 DEBUG 21:48:01 DEBUG ] 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 DEBUG 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG ] 21:48:01 DEBUG 21:48:01 DEBUG ] 21:48:01 DEBUG ] 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 DEBUG [...] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 DEBUG 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:48:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG }, 21:48:01 DEBUG { 21:48:01 DEBUG "value": [ 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "data": { 21:48:01 DEBUG "0" 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG ] 21:48:01 DEBUG "result": [ 21:48:01 DEBUG } 21:48:01 DEBUG { 21:48:01 DEBUG ] 21:48:01 DEBUG "metric": { 21:48:01 DEBUG } 21:48:01 DEBUG "pod": "ds-cts-0" 21:48:01 DEBUG } 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "ds-cts-2" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "pod": "ds-cts-2" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0.000018848208554764167" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "pod": "ds-cts-2" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "pod": "ds-cts-0" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "ds-idrepo-0" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "pod": "ds-idrepo-2" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "ds-idrepo-0" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "671.5333333333333" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "instance": "10.142.0.100:9100" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "instance": "10.142.0.99:9100" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0.0003333333333333262" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "13617684480" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "pod": "ds-cts-2" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "pod": "ds-idrepo-2" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "ds-idrepo-0" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "671.5333333333333" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "instance": "10.142.0.100:9100" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "instance": "10.142.0.99:9100" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "20298951.566666666" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689281" 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "instance": "10.142.0.100:9100" 21:48:01 DEBUG "instance": "10.142.0.100:9100" 21:48:01 DEBUG }, 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "instance": "10.142.0.99:9100" 21:48:01 DEBUG "instance": "10.142.0.99:9100" 21:48:01 DEBUG }, 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0.00030000000000048505" 21:48:01 DEBUG "20299339.066666666" 21:48:01 DEBUG ] 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "pod": "ds-idrepo-2" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "pod": "ds-idrepo-0" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "671.5666666666666" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689281" 21:48:01 INFO [http_cmd]: http status code OK 21:48:01 DEBUG --- status code --- 21:48:01 DEBUG http status code is 200 (expected 200) 21:48:01 DEBUG --- http response --- 21:48:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:48:01 DEBUG ----- output ----- 21:48:01 DEBUG { 21:48:01 DEBUG "status": "success", 21:48:01 DEBUG "data": { 21:48:01 DEBUG "resultType": "vector", 21:48:01 DEBUG "result": [ 21:48:01 DEBUG { 21:48:01 DEBUG "metric": { 21:48:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG [...] 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG 21:48:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:48:01 DEBUG }, 21:48:01 DEBUG "value": [ 21:48:01 DEBUG 1686689281, 21:48:01 DEBUG "0.23050568945409688" 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG ] 21:48:01 DEBUG } 21:48:01 DEBUG } 21:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689282" 21:48:02 INFO [http_cmd]: http status code OK 21:48:02 DEBUG --- status code --- 21:48:02 DEBUG http status code is 200 (expected 200) 21:48:02 DEBUG --- http response --- 21:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:48:02 DEBUG ----- output ----- 21:48:02 DEBUG { 21:48:02 DEBUG "status": "success", 21:48:02 DEBUG "data": { 21:48:02 DEBUG "resultType": "vector", 21:48:02 DEBUG "result": [ 21:48:02 DEBUG { 21:48:02 DEBUG "metric": { 21:48:02 DEBUG "pod": "ds-cts-2" 21:48:02 DEBUG }, 21:48:02 DEBUG "value": [ 21:48:02 DEBUG 21:48:02 DEBUG 21:48:02 DEBUG 21:48:02 DEBUG [...] 21:48:02 DEBUG 21:48:02 DEBUG 21:48:02 DEBUG 21:48:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:48:02 DEBUG }, 21:48:02 DEBUG "value": [ 21:48:02 DEBUG 1686689282, 21:48:02 DEBUG "0" 21:48:02 DEBUG ] 21:48:02 DEBUG } 21:48:02 DEBUG ] 21:48:02 DEBUG } 21:48:02 DEBUG } 21:48:16 INFO 21:48:16 INFO [loop_until]: kubectl --namespace=xlou top pods 21:48:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:48:16 INFO [loop_until]: OK (rc = 0) 21:48:16 DEBUG --- stdout --- 21:48:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 422Mi ds-cts-2 9m 392Mi ds-idrepo-0 3435m 13802Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3338m 4851Mi idm-58dc667486-rgz9j 2626m 5090Mi lodemon-6455d5ff7c-zzndq 2m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 203m 638Mi 21:48:16 DEBUG --- stderr --- 21:48:16 DEBUG 21:48:19 INFO 21:48:19 INFO [loop_until]: kubectl --namespace=xlou top node 21:48:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:48:19 INFO [loop_until]: OK (rc = 0) 21:48:19 DEBUG --- stdout --- 21:48:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3325m 20% 5802Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 73m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 69m 0% 6816Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2798m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 69m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 437m 2% 3158Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1194Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 66m 0% 13709Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3595m 22% 14520Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14586Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 267m 1% 1951Mi 3% 21:48:19 DEBUG --- stderr --- 21:48:19 DEBUG 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG { 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG { 21:49:01 DEBUG { 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "data": { 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG "data": { 21:49:01 DEBUG "data": { 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG "result": [ 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG "result": [ 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG { 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "data": { 21:49:01 DEBUG { 21:49:01 DEBUG { 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "pod": "ds-cts-0" 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "pod": "ds-cts-0" 21:49:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:49:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:49:01 DEBUG }, 21:49:01 DEBUG "result": [ 21:49:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:49:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:49:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:49:01 DEBUG }, 21:49:01 DEBUG }, 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG { 21:49:01 DEBUG }, 21:49:01 DEBUG }, 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG "value": [ 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "value": [ 21:49:01 DEBUG "value": [ 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG }, 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG [...] 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG [...] 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "pod": "ds-idrepo-0" 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "pod": "ds-cts-2" 21:49:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:49:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:49:01 DEBUG }, 21:49:01 DEBUG 21:49:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:49:01 DEBUG "pod": "am-7849cf7bdb-lzlql" 21:49:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:49:01 DEBUG }, 21:49:01 DEBUG }, 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG }, 21:49:01 DEBUG }, 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG "value": [ 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG 21:49:01 DEBUG "value": [ 21:49:01 DEBUG "value": [ 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "647.6333333333333" 21:49:01 DEBUG "pod": "am-7849cf7bdb-lzlql" 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "0" 21:49:01 DEBUG "0" 21:49:01 DEBUG "0" 21:49:01 DEBUG ] 21:49:01 DEBUG }, 21:49:01 DEBUG "0" 21:49:01 DEBUG "0" 21:49:01 DEBUG "0.2" 21:49:01 DEBUG ] 21:49:01 DEBUG ] 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG "value": [ 21:49:01 DEBUG ] 21:49:01 DEBUG ] 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG ] 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG "0" 21:49:01 DEBUG ] 21:49:01 DEBUG ] 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "pod": "ds-cts-0" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "pod": "ds-idrepo-0" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "0" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "pod": "ds-idrepo-2" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "pod": "ds-idrepo-0" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "647.6333333333333" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "instance": "10.142.0.100:9100" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "instance": "10.142.0.99:9100" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "0" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "13616353280" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "0" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "0" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "pod": "ds-idrepo-2" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "pod": "ds-idrepo-0" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "647.5666666666666" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "instance": "10.142.0.100:9100" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "instance": "10.142.0.99:9100" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "21006564.5" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "instance": "10.142.0.100:9100" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "instance": "10.142.0.99:9100" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "21008258.966666665" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "instance": "10.142.0.100:9100" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "instance": "10.142.0.99:9100" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "0.0004999999999995453" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689341" 21:49:01 INFO [http_cmd]: http status code OK 21:49:01 DEBUG --- status code --- 21:49:01 DEBUG http status code is 200 (expected 200) 21:49:01 DEBUG --- http response --- 21:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:49:01 DEBUG ----- output ----- 21:49:01 DEBUG { 21:49:01 DEBUG "status": "success", 21:49:01 DEBUG "data": { 21:49:01 DEBUG "resultType": "vector", 21:49:01 DEBUG "result": [ 21:49:01 DEBUG { 21:49:01 DEBUG "metric": { 21:49:01 DEBUG "pod": "ds-idrepo-2" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG [...] 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG 21:49:01 DEBUG "pod": "ds-idrepo-0" 21:49:01 DEBUG }, 21:49:01 DEBUG "value": [ 21:49:01 DEBUG 1686689341, 21:49:01 DEBUG "647.5666666666666" 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG ] 21:49:01 DEBUG } 21:49:01 DEBUG } 21:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689342" 21:49:02 INFO [http_cmd]: http status code OK 21:49:02 DEBUG --- status code --- 21:49:02 DEBUG http status code is 200 (expected 200) 21:49:02 DEBUG --- http response --- 21:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:49:02 DEBUG ----- output ----- 21:49:02 DEBUG { 21:49:02 DEBUG "status": "success", 21:49:02 DEBUG "data": { 21:49:02 DEBUG "resultType": "vector", 21:49:02 DEBUG "result": [ 21:49:02 DEBUG { 21:49:02 DEBUG "metric": { 21:49:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:49:02 DEBUG }, 21:49:02 DEBUG "value": [ 21:49:02 DEBUG 21:49:02 DEBUG 21:49:02 DEBUG 21:49:02 DEBUG [...] 21:49:02 DEBUG 21:49:02 DEBUG 21:49:02 DEBUG 21:49:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:49:02 DEBUG }, 21:49:02 DEBUG "value": [ 21:49:02 DEBUG 1686689342, 21:49:02 DEBUG "0.23134124245382856" 21:49:02 DEBUG ] 21:49:02 DEBUG } 21:49:02 DEBUG ] 21:49:02 DEBUG } 21:49:02 DEBUG } 21:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689342" 21:49:02 INFO [http_cmd]: http status code OK 21:49:02 DEBUG --- status code --- 21:49:02 DEBUG http status code is 200 (expected 200) 21:49:02 DEBUG --- http response --- 21:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:49:02 DEBUG ----- output ----- 21:49:02 DEBUG { 21:49:02 DEBUG "status": "success", 21:49:02 DEBUG "data": { 21:49:02 DEBUG "resultType": "vector", 21:49:02 DEBUG "result": [ 21:49:02 DEBUG { 21:49:02 DEBUG "metric": { 21:49:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:49:02 DEBUG }, 21:49:02 DEBUG "value": [ 21:49:02 DEBUG 21:49:02 DEBUG 21:49:02 DEBUG 21:49:02 DEBUG [...] 21:49:02 DEBUG 21:49:02 DEBUG 21:49:02 DEBUG 21:49:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:49:02 DEBUG }, 21:49:02 DEBUG "value": [ 21:49:02 DEBUG 1686689342, 21:49:02 DEBUG "0" 21:49:02 DEBUG ] 21:49:02 DEBUG } 21:49:02 DEBUG ] 21:49:02 DEBUG } 21:49:02 DEBUG } 21:49:16 INFO 21:49:16 INFO [loop_until]: kubectl --namespace=xlou top pods 21:49:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:49:16 INFO [loop_until]: OK (rc = 0) 21:49:16 DEBUG --- stdout --- 21:49:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3371m 13803Mi ds-idrepo-1 19m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2962m 4852Mi idm-58dc667486-rgz9j 2588m 5090Mi lodemon-6455d5ff7c-zzndq 2m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 211m 638Mi 21:49:16 DEBUG --- stderr --- 21:49:16 DEBUG 21:49:20 INFO 21:49:20 INFO [loop_until]: kubectl --namespace=xlou top node 21:49:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:49:20 INFO [loop_until]: OK (rc = 0) 21:49:20 DEBUG --- stdout --- 21:49:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3275m 20% 5800Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6852Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2738m 17% 6070Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 68m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 448m 2% 3152Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1195Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13703Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3393m 21% 14494Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 74m 0% 14585Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 270m 1% 1952Mi 3% 21:49:20 DEBUG --- stderr --- 21:49:20 DEBUG 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:50:01 DEBUG "pod": "ds-cts-2" 21:50:01 DEBUG }, 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:50:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:50:01 DEBUG }, 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0" 21:50:01 DEBUG "0.00002376245274381103" 21:50:01 DEBUG ] 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "ds-cts-2" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "ds-cts-0" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "ds-cts-2" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "ds-idrepo-2" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "ds-idrepo-0" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "651.3" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG } 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "ds-cts-0" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "ds-idrepo-0" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "651.3" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0.2" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "ds-cts-0" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "ds-idrepo-0" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "instance": "10.142.0.100:9100" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "instance": "10.142.0.99:9100" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "13614350336" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "ds-cts-2" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "pod": "ds-idrepo-2" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "pod": "ds-idrepo-0" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "651.3333333333334" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "instance": "10.142.0.100:9100" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "instance": "10.142.0.99:9100" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "18038900.799999997" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "instance": "10.142.0.100:9100" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "instance": "10.142.0.99:9100" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "18040952.9" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689401" 21:50:01 INFO [http_cmd]: http status code OK 21:50:01 DEBUG --- status code --- 21:50:01 DEBUG http status code is 200 (expected 200) 21:50:01 DEBUG --- http response --- 21:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:50:01 DEBUG ----- output ----- 21:50:01 DEBUG { 21:50:01 DEBUG "status": "success", 21:50:01 DEBUG "data": { 21:50:01 DEBUG "resultType": "vector", 21:50:01 DEBUG "result": [ 21:50:01 DEBUG { 21:50:01 DEBUG "metric": { 21:50:01 DEBUG "instance": "10.142.0.100:9100" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG [...] 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG 21:50:01 DEBUG "instance": "10.142.0.99:9100" 21:50:01 DEBUG }, 21:50:01 DEBUG "value": [ 21:50:01 DEBUG 1686689401, 21:50:01 DEBUG "0.0005666666666665302" 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG ] 21:50:01 DEBUG } 21:50:01 DEBUG } 21:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689402" 21:50:02 INFO [http_cmd]: http status code OK 21:50:02 DEBUG --- status code --- 21:50:02 DEBUG http status code is 200 (expected 200) 21:50:02 DEBUG --- http response --- 21:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:50:02 DEBUG ----- output ----- 21:50:02 DEBUG { 21:50:02 DEBUG "status": "success", 21:50:02 DEBUG "data": { 21:50:02 DEBUG "resultType": "vector", 21:50:02 DEBUG "result": [ 21:50:02 DEBUG { 21:50:02 DEBUG "metric": { 21:50:02 DEBUG "pod": "ds-idrepo-2" 21:50:02 DEBUG }, 21:50:02 DEBUG "value": [ 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG [...] 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG "pod": "ds-idrepo-0" 21:50:02 DEBUG }, 21:50:02 DEBUG "value": [ 21:50:02 DEBUG 1686689402, 21:50:02 DEBUG "651.3333333333334" 21:50:02 DEBUG ] 21:50:02 DEBUG } 21:50:02 DEBUG ] 21:50:02 DEBUG } 21:50:02 DEBUG } 21:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689402" 21:50:02 INFO [http_cmd]: http status code OK 21:50:02 DEBUG --- status code --- 21:50:02 DEBUG http status code is 200 (expected 200) 21:50:02 DEBUG --- http response --- 21:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:50:02 DEBUG ----- output ----- 21:50:02 DEBUG { 21:50:02 DEBUG "status": "success", 21:50:02 DEBUG "data": { 21:50:02 DEBUG "resultType": "vector", 21:50:02 DEBUG "result": [ 21:50:02 DEBUG { 21:50:02 DEBUG "metric": { 21:50:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:50:02 DEBUG }, 21:50:02 DEBUG "value": [ 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG [...] 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:50:02 DEBUG }, 21:50:02 DEBUG "value": [ 21:50:02 DEBUG 1686689402, 21:50:02 DEBUG "0.21696078981397726" 21:50:02 DEBUG ] 21:50:02 DEBUG } 21:50:02 DEBUG ] 21:50:02 DEBUG } 21:50:02 DEBUG } 21:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689402" 21:50:02 INFO [http_cmd]: http status code OK 21:50:02 DEBUG --- status code --- 21:50:02 DEBUG http status code is 200 (expected 200) 21:50:02 DEBUG --- http response --- 21:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 21:50:02 DEBUG ----- output ----- 21:50:02 DEBUG { 21:50:02 DEBUG "status": "success", 21:50:02 DEBUG "data": { 21:50:02 DEBUG "resultType": "vector", 21:50:02 DEBUG "result": [ 21:50:02 DEBUG { 21:50:02 DEBUG "metric": { 21:50:02 DEBUG "pod": "ds-cts-2" 21:50:02 DEBUG }, 21:50:02 DEBUG "value": [ 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG [...] 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG 21:50:02 DEBUG "pod": "lodemon-6455d5ff7c-zzndq" 21:50:02 DEBUG }, 21:50:02 DEBUG "value": [ 21:50:02 DEBUG 1686689402, 21:50:02 DEBUG "0" 21:50:02 DEBUG ] 21:50:02 DEBUG } 21:50:02 DEBUG ] 21:50:02 DEBUG } 21:50:02 DEBUG } 21:50:16 INFO 21:50:16 INFO [loop_until]: kubectl --namespace=xlou top pods 21:50:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:50:16 INFO [loop_until]: OK (rc = 0) 21:50:16 DEBUG --- stdout --- 21:50:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3591m 13803Mi ds-idrepo-1 19m 13821Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3198m 4854Mi idm-58dc667486-rgz9j 2962m 5090Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 215m 638Mi 21:50:16 DEBUG --- stderr --- 21:50:16 DEBUG 21:50:20 INFO 21:50:20 INFO [loop_until]: kubectl --namespace=xlou top node 21:50:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:50:20 INFO [loop_until]: OK (rc = 0) 21:50:20 DEBUG --- stdout --- 21:50:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3204m 20% 5801Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6816Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 3087m 19% 6075Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 436m 2% 3156Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1475Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13706Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3517m 22% 14500Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 76m 0% 14587Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 273m 1% 1953Mi 3% 21:50:20 DEBUG --- stderr --- 21:50:20 DEBUG 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG --- status code --- 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG --- status code --- 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- status code --- 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:51:01 DEBUG --- http response --- 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:51:01 DEBUG { 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG { 21:51:01 DEBUG { 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG { 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG { 21:51:01 DEBUG { 21:51:01 DEBUG "data": { 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG { 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG "data": { 21:51:01 DEBUG "data": { 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "data": { 21:51:01 DEBUG "status": "success", 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "data": { 21:51:01 DEBUG "data": { 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "data": { 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG { 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG { 21:51:01 DEBUG { 21:51:01 DEBUG { 21:51:01 DEBUG { 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "metric": { 21:51:01 DEBUG { 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG { 21:51:01 DEBUG { 21:51:01 DEBUG "pod": "ds-cts-0" 21:51:01 DEBUG "metric": { 21:51:01 DEBUG { 21:51:01 DEBUG "data": { 21:51:01 DEBUG "data": { 21:51:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:51:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG }, 21:51:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG }, 21:51:01 DEBUG }, 21:51:01 DEBUG "pod": "ds-idrepo-2" 21:51:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:51:01 DEBUG "value": [ 21:51:01 DEBUG }, 21:51:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "value": [ 21:51:01 DEBUG "value": [ 21:51:01 DEBUG }, 21:51:01 DEBUG }, 21:51:01 DEBUG 21:51:01 DEBUG "value": [ 21:51:01 DEBUG }, 21:51:01 DEBUG { 21:51:01 DEBUG { 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "value": [ 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "value": [ 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "pod": "ds-cts-0" 21:51:01 DEBUG "pod": "ds-cts-0" 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG }, 21:51:01 DEBUG }, 21:51:01 DEBUG [...] 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG "value": [ 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "pod": "ds-cts-2" 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:51:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG }, 21:51:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG [...] 21:51:01 DEBUG }, 21:51:01 DEBUG }, 21:51:01 DEBUG "pod": "ds-idrepo-0" 21:51:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:51:01 DEBUG "value": [ 21:51:01 DEBUG }, 21:51:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "value": [ 21:51:01 DEBUG "value": [ 21:51:01 DEBUG }, 21:51:01 DEBUG }, 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG }, 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG "value": [ 21:51:01 DEBUG "0" 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "0.2" 21:51:01 DEBUG "0" 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG ] 21:51:01 DEBUG "0.00593360097956406" 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "pod": "ds-idrepo-0" 21:51:01 DEBUG "pod": "ds-idrepo-0" 21:51:01 DEBUG ] 21:51:01 DEBUG ] 21:51:01 DEBUG "596.7" 21:51:01 DEBUG "0" 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG "4427.360243202162" 21:51:01 DEBUG }, 21:51:01 DEBUG }, 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG ] 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG ] 21:51:01 DEBUG "value": [ 21:51:01 DEBUG "value": [ 21:51:01 DEBUG ] 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG } 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG ] 21:51:01 DEBUG "596.7333333333333" 21:51:01 DEBUG "0" 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG { 21:51:01 DEBUG ] 21:51:01 DEBUG ] 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG "data": { 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "result": [ 21:51:01 DEBUG { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "0" 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689461" 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689461" 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- http response --- 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG { 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:51:01 DEBUG "data": { 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG { 21:51:01 DEBUG "result": [ 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG { 21:51:01 DEBUG "data": { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:51:01 DEBUG "result": [ 21:51:01 DEBUG }, 21:51:01 DEBUG { 21:51:01 DEBUG "value": [ 21:51:01 DEBUG "metric": { 21:51:01 DEBUG 21:51:01 DEBUG "instance": "10.142.0.100:9100" 21:51:01 DEBUG 21:51:01 DEBUG }, 21:51:01 DEBUG 21:51:01 DEBUG "value": [ 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:51:01 DEBUG 21:51:01 DEBUG }, 21:51:01 DEBUG 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "instance": "10.142.0.99:9100" 21:51:01 DEBUG "13615202304" 21:51:01 DEBUG }, 21:51:01 DEBUG ] 21:51:01 DEBUG "value": [ 21:51:01 DEBUG } 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG ] 21:51:01 DEBUG "0" 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG { 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "data": { 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "result": [ 21:51:01 DEBUG { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "0" 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG { 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "data": { 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "result": [ 21:51:01 DEBUG { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "0" 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689461" 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG { 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "data": { 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "result": [ 21:51:01 DEBUG { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "pod": "ds-idrepo-2" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "pod": "ds-idrepo-0" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "596.6666666666666" 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689461" 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG { 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "data": { 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "result": [ 21:51:01 DEBUG { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "instance": "10.142.0.100:9100" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "instance": "10.142.0.99:9100" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "18489223.9" 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689461" 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG { 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "data": { 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "result": [ 21:51:01 DEBUG { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "instance": "10.142.0.100:9100" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "instance": "10.142.0.99:9100" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "18489598.433333334" 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689461" 21:51:01 INFO [http_cmd]: http status code OK 21:51:01 DEBUG --- status code --- 21:51:01 DEBUG http status code is 200 (expected 200) 21:51:01 DEBUG --- http response --- 21:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:51:01 DEBUG ----- output ----- 21:51:01 DEBUG { 21:51:01 DEBUG "status": "success", 21:51:01 DEBUG "data": { 21:51:01 DEBUG "resultType": "vector", 21:51:01 DEBUG "result": [ 21:51:01 DEBUG { 21:51:01 DEBUG "metric": { 21:51:01 DEBUG "instance": "10.142.0.100:9100" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG [...] 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG 21:51:01 DEBUG "instance": "10.142.0.99:9100" 21:51:01 DEBUG }, 21:51:01 DEBUG "value": [ 21:51:01 DEBUG 1686689461, 21:51:01 DEBUG "0.00040000000000001514" 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG ] 21:51:01 DEBUG } 21:51:01 DEBUG } 21:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689462" 21:51:02 INFO [http_cmd]: http status code OK 21:51:02 DEBUG --- status code --- 21:51:02 DEBUG http status code is 200 (expected 200) 21:51:02 DEBUG --- http response --- 21:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:51:02 DEBUG ----- output ----- 21:51:02 DEBUG { 21:51:02 DEBUG "status": "success", 21:51:02 DEBUG "data": { 21:51:02 DEBUG "resultType": "vector", 21:51:02 DEBUG "result": [ 21:51:02 DEBUG { 21:51:02 DEBUG "metric": { 21:51:02 DEBUG "pod": "ds-idrepo-2" 21:51:02 DEBUG }, 21:51:02 DEBUG "value": [ 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG [...] 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG "pod": "ds-idrepo-0" 21:51:02 DEBUG }, 21:51:02 DEBUG "value": [ 21:51:02 DEBUG 1686689462, 21:51:02 DEBUG "596.6333333333333" 21:51:02 DEBUG ] 21:51:02 DEBUG } 21:51:02 DEBUG ] 21:51:02 DEBUG } 21:51:02 DEBUG } 21:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689462" 21:51:02 INFO [http_cmd]: http status code OK 21:51:02 DEBUG --- status code --- 21:51:02 DEBUG http status code is 200 (expected 200) 21:51:02 DEBUG --- http response --- 21:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:51:02 DEBUG ----- output ----- 21:51:02 DEBUG { 21:51:02 DEBUG "status": "success", 21:51:02 DEBUG "data": { 21:51:02 DEBUG "resultType": "vector", 21:51:02 DEBUG "result": [ 21:51:02 DEBUG { 21:51:02 DEBUG "metric": { 21:51:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:51:02 DEBUG }, 21:51:02 DEBUG "value": [ 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG [...] 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:51:02 DEBUG }, 21:51:02 DEBUG "value": [ 21:51:02 DEBUG 1686689462, 21:51:02 DEBUG "0.22362548286668435" 21:51:02 DEBUG ] 21:51:02 DEBUG } 21:51:02 DEBUG ] 21:51:02 DEBUG } 21:51:02 DEBUG } 21:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689462" 21:51:02 INFO [http_cmd]: http status code OK 21:51:02 DEBUG --- status code --- 21:51:02 DEBUG http status code is 200 (expected 200) 21:51:02 DEBUG --- http response --- 21:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:51:02 DEBUG ----- output ----- 21:51:02 DEBUG { 21:51:02 DEBUG "status": "success", 21:51:02 DEBUG "data": { 21:51:02 DEBUG "resultType": "vector", 21:51:02 DEBUG "result": [ 21:51:02 DEBUG { 21:51:02 DEBUG "metric": { 21:51:02 DEBUG "pod": "ds-cts-2" 21:51:02 DEBUG }, 21:51:02 DEBUG "value": [ 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG [...] 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG 21:51:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:51:02 DEBUG }, 21:51:02 DEBUG "value": [ 21:51:02 DEBUG 1686689462, 21:51:02 DEBUG "0" 21:51:02 DEBUG ] 21:51:02 DEBUG } 21:51:02 DEBUG ] 21:51:02 DEBUG } 21:51:02 DEBUG } 21:51:16 INFO 21:51:16 INFO [loop_until]: kubectl --namespace=xlou top pods 21:51:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:51:16 INFO [loop_until]: OK (rc = 0) 21:51:16 DEBUG --- stdout --- 21:51:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 10m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 423Mi ds-cts-2 8m 392Mi ds-idrepo-0 3439m 13804Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3322m 4856Mi idm-58dc667486-rgz9j 2648m 5091Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 211m 639Mi 21:51:16 DEBUG --- stderr --- 21:51:16 DEBUG 21:51:20 INFO 21:51:20 INFO [loop_until]: kubectl --namespace=xlou top node 21:51:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:51:20 INFO [loop_until]: OK (rc = 0) 21:51:20 DEBUG --- stdout --- 21:51:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3227m 20% 5806Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6813Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2741m 17% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 69m 0% 6945Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 451m 2% 3143Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1476Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 59m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1195Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 66m 0% 13707Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3479m 21% 14499Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14591Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 276m 1% 1953Mi 3% 21:51:20 DEBUG --- stderr --- 21:51:20 DEBUG 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG { 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "result": [ 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "result": [ 21:52:01 DEBUG { 21:52:01 DEBUG "result": [ 21:52:01 DEBUG { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "pod": "ds-cts-0" 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:52:01 DEBUG }, 21:52:01 DEBUG "pod": "ds-cts-0" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG "pod": "ds-idrepo-0" 21:52:01 DEBUG 21:52:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:52:01 DEBUG }, 21:52:01 DEBUG "pod": "ds-idrepo-0" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "559.5333333333333" 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "0" 21:52:01 DEBUG ] 21:52:01 DEBUG "0" 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG { 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG "data": { 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG "result": [ 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:52:01 DEBUG { 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG "metric": { 21:52:01 DEBUG { 21:52:01 DEBUG { 21:52:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG }, 21:52:01 DEBUG "data": { 21:52:01 DEBUG "data": { 21:52:01 DEBUG "value": [ 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG 21:52:01 DEBUG "result": [ 21:52:01 DEBUG "result": [ 21:52:01 DEBUG 21:52:01 DEBUG { 21:52:01 DEBUG { 21:52:01 DEBUG 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG [...] 21:52:01 DEBUG "pod": "ds-cts-0" 21:52:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:52:01 DEBUG 21:52:01 DEBUG }, 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG }, 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG 21:52:01 DEBUG "value": [ 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG "value": [ 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:52:01 DEBUG 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG }, 21:52:01 DEBUG 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:52:01 DEBUG 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:52:01 DEBUG "value": [ 21:52:01 DEBUG [...] 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG [...] 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG 21:52:01 DEBUG { 21:52:01 DEBUG 21:52:01 DEBUG { 21:52:01 DEBUG { 21:52:01 DEBUG { 21:52:01 DEBUG "0.2" 21:52:01 DEBUG 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG ] 21:52:01 DEBUG 21:52:01 DEBUG "data": { 21:52:01 DEBUG 21:52:01 DEBUG "data": { 21:52:01 DEBUG "data": { 21:52:01 DEBUG "data": { 21:52:01 DEBUG } 21:52:01 DEBUG "pod": "ds-cts-2" 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG ] 21:52:01 DEBUG }, 21:52:01 DEBUG "result": [ 21:52:01 DEBUG }, 21:52:01 DEBUG "result": [ 21:52:01 DEBUG "result": [ 21:52:01 DEBUG "result": [ 21:52:01 DEBUG } 21:52:01 DEBUG "value": [ 21:52:01 DEBUG { 21:52:01 DEBUG "value": [ 21:52:01 DEBUG { 21:52:01 DEBUG { 21:52:01 DEBUG { 21:52:01 DEBUG } 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "metric": { 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "0" 21:52:01 DEBUG "pod": "ds-idrepo-2" 21:52:01 DEBUG "0" 21:52:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:52:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:52:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:52:01 DEBUG ] 21:52:01 DEBUG }, 21:52:01 DEBUG ] 21:52:01 DEBUG }, 21:52:01 DEBUG }, 21:52:01 DEBUG }, 21:52:01 DEBUG } 21:52:01 DEBUG "value": [ 21:52:01 DEBUG } 21:52:01 DEBUG "value": [ 21:52:01 DEBUG "value": [ 21:52:01 DEBUG "value": [ 21:52:01 DEBUG ] 21:52:01 DEBUG 21:52:01 DEBUG ] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG } 21:52:01 DEBUG 21:52:01 DEBUG } 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG } 21:52:01 DEBUG 21:52:01 DEBUG } 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG [...] 21:52:01 DEBUG [...] 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG "pod": "ds-idrepo-0" 21:52:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:52:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:52:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:52:01 DEBUG }, 21:52:01 DEBUG }, 21:52:01 DEBUG }, 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG "value": [ 21:52:01 DEBUG "value": [ 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "559.5666666666666" 21:52:01 DEBUG "0" 21:52:01 DEBUG "0" 21:52:01 DEBUG "0" 21:52:01 DEBUG ] 21:52:01 DEBUG ] 21:52:01 DEBUG ] 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG ] 21:52:01 DEBUG ] 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689521" 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689521" 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:52:01 DEBUG { 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG { 21:52:01 DEBUG "data": { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "result": [ 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG { 21:52:01 DEBUG "result": [ 21:52:01 DEBUG "metric": { 21:52:01 DEBUG { 21:52:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:52:01 DEBUG "metric": { 21:52:01 DEBUG }, 21:52:01 DEBUG "instance": "10.142.0.100:9100" 21:52:01 DEBUG "value": [ 21:52:01 DEBUG }, 21:52:01 DEBUG 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:52:01 DEBUG 21:52:01 DEBUG }, 21:52:01 DEBUG "instance": "10.142.0.99:9100" 21:52:01 DEBUG "value": [ 21:52:01 DEBUG }, 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG "13616549888" 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG ] 21:52:01 DEBUG "0" 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "result": [ 21:52:01 DEBUG { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "0" 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "result": [ 21:52:01 DEBUG { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "0" 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689521" 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "result": [ 21:52:01 DEBUG { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "pod": "ds-idrepo-2" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG "pod": "ds-idrepo-0" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "559.5666666666666" 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689521" 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "result": [ 21:52:01 DEBUG { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "instance": "10.142.0.100:9100" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG "instance": "10.142.0.99:9100" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "17135426.6" 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689521" 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "result": [ 21:52:01 DEBUG { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "instance": "10.142.0.100:9100" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG "instance": "10.142.0.99:9100" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "17137199.9" 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689521" 21:52:01 INFO [http_cmd]: http status code OK 21:52:01 DEBUG --- status code --- 21:52:01 DEBUG http status code is 200 (expected 200) 21:52:01 DEBUG --- http response --- 21:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:52:01 DEBUG ----- output ----- 21:52:01 DEBUG { 21:52:01 DEBUG "status": "success", 21:52:01 DEBUG "data": { 21:52:01 DEBUG "resultType": "vector", 21:52:01 DEBUG "result": [ 21:52:01 DEBUG { 21:52:01 DEBUG "metric": { 21:52:01 DEBUG "instance": "10.142.0.100:9100" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG [...] 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG 21:52:01 DEBUG "instance": "10.142.0.99:9100" 21:52:01 DEBUG }, 21:52:01 DEBUG "value": [ 21:52:01 DEBUG 1686689521, 21:52:01 DEBUG "0.00026666666666604516" 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG ] 21:52:01 DEBUG } 21:52:01 DEBUG } 21:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689522" 21:52:02 INFO [http_cmd]: http status code OK 21:52:02 DEBUG --- status code --- 21:52:02 DEBUG http status code is 200 (expected 200) 21:52:02 DEBUG --- http response --- 21:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:52:02 DEBUG ----- output ----- 21:52:02 DEBUG { 21:52:02 DEBUG "status": "success", 21:52:02 DEBUG "data": { 21:52:02 DEBUG "resultType": "vector", 21:52:02 DEBUG "result": [ 21:52:02 DEBUG { 21:52:02 DEBUG "metric": { 21:52:02 DEBUG "pod": "ds-idrepo-2" 21:52:02 DEBUG }, 21:52:02 DEBUG "value": [ 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG [...] 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG "pod": "ds-idrepo-0" 21:52:02 DEBUG }, 21:52:02 DEBUG "value": [ 21:52:02 DEBUG 1686689522, 21:52:02 DEBUG "559.6" 21:52:02 DEBUG ] 21:52:02 DEBUG } 21:52:02 DEBUG ] 21:52:02 DEBUG } 21:52:02 DEBUG } 21:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689522" 21:52:02 INFO [http_cmd]: http status code OK 21:52:02 DEBUG --- status code --- 21:52:02 DEBUG http status code is 200 (expected 200) 21:52:02 DEBUG --- http response --- 21:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:52:02 DEBUG ----- output ----- 21:52:02 DEBUG { 21:52:02 DEBUG "status": "success", 21:52:02 DEBUG "data": { 21:52:02 DEBUG "resultType": "vector", 21:52:02 DEBUG "result": [ 21:52:02 DEBUG { 21:52:02 DEBUG "metric": { 21:52:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:52:02 DEBUG }, 21:52:02 DEBUG "value": [ 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG [...] 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:52:02 DEBUG }, 21:52:02 DEBUG "value": [ 21:52:02 DEBUG 1686689522, 21:52:02 DEBUG "0.21749969888061094" 21:52:02 DEBUG ] 21:52:02 DEBUG } 21:52:02 DEBUG ] 21:52:02 DEBUG } 21:52:02 DEBUG } 21:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689522" 21:52:02 INFO [http_cmd]: http status code OK 21:52:02 DEBUG --- status code --- 21:52:02 DEBUG http status code is 200 (expected 200) 21:52:02 DEBUG --- http response --- 21:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:52:02 DEBUG ----- output ----- 21:52:02 DEBUG { 21:52:02 DEBUG "status": "success", 21:52:02 DEBUG "data": { 21:52:02 DEBUG "resultType": "vector", 21:52:02 DEBUG "result": [ 21:52:02 DEBUG { 21:52:02 DEBUG "metric": { 21:52:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:52:02 DEBUG }, 21:52:02 DEBUG "value": [ 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG [...] 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG 21:52:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:52:02 DEBUG }, 21:52:02 DEBUG "value": [ 21:52:02 DEBUG 1686689522, 21:52:02 DEBUG "0" 21:52:02 DEBUG ] 21:52:02 DEBUG } 21:52:02 DEBUG ] 21:52:02 DEBUG } 21:52:02 DEBUG } 21:52:17 INFO 21:52:17 INFO [loop_until]: kubectl --namespace=xlou top pods 21:52:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:52:17 INFO [loop_until]: OK (rc = 0) 21:52:17 DEBUG --- stdout --- 21:52:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3330m 13826Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2909m 4857Mi idm-58dc667486-rgz9j 2630m 5091Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 205m 639Mi 21:52:17 DEBUG --- stderr --- 21:52:17 DEBUG 21:52:20 INFO 21:52:20 INFO [loop_until]: kubectl --namespace=xlou top node 21:52:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:52:20 INFO [loop_until]: OK (rc = 0) 21:52:20 DEBUG --- stdout --- 21:52:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3167m 19% 5804Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6820Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2749m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 438m 2% 3145Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 85m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1194Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13710Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3505m 22% 14496Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14587Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 265m 1% 1954Mi 3% 21:52:20 DEBUG --- stderr --- 21:52:20 DEBUG 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "2.323763633569656" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "pod": "ds-cts-0" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "pod": "ds-idrepo-0" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "0" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG --- http response --- 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG { 21:53:01 DEBUG { 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "data": { 21:53:01 DEBUG "data": { 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG "result": [ 21:53:01 DEBUG "result": [ 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG { 21:53:01 DEBUG { 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:53:01 DEBUG "pod": "ds-cts-0" 21:53:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:53:01 DEBUG "pod": "ds-cts-0" 21:53:01 DEBUG }, 21:53:01 DEBUG }, 21:53:01 DEBUG }, 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG "value": [ 21:53:01 DEBUG "value": [ 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG [...] 21:53:01 DEBUG [...] 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:53:01 DEBUG "pod": "ds-cts-2" 21:53:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:53:01 DEBUG "pod": "ds-idrepo-0" 21:53:01 DEBUG }, 21:53:01 DEBUG }, 21:53:01 DEBUG }, 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG "value": [ 21:53:01 DEBUG "value": [ 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "176960627.39164695" 21:53:01 DEBUG "0" 21:53:01 DEBUG "0.2" 21:53:01 DEBUG "573.2" 21:53:01 DEBUG ] 21:53:01 DEBUG ] 21:53:01 DEBUG ] 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG ] 21:53:01 DEBUG ] 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG } 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "pod": "ds-idrepo-2" 21:53:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:53:01 DEBUG }, 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "pod": "ds-idrepo-0" 21:53:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:53:01 DEBUG }, 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "573.2" 21:53:01 DEBUG "0" 21:53:01 DEBUG ] 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "0" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "0" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "13617897472" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "instance": "10.142.0.100:9100" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "instance": "10.142.0.99:9100" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "0" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "0" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "0" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "pod": "ds-idrepo-2" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "pod": "ds-idrepo-0" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "573.2333333333333" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689581" 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "instance": "10.142.0.100:9100" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "instance": "10.142.0.99:9100" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "18800821.766666666" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "instance": "10.142.0.100:9100" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "instance": "10.142.0.99:9100" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "0.00040000000000001514" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689581" 21:53:01 INFO [http_cmd]: http status code OK 21:53:01 DEBUG --- status code --- 21:53:01 DEBUG http status code is 200 (expected 200) 21:53:01 DEBUG --- http response --- 21:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:53:01 DEBUG ----- output ----- 21:53:01 DEBUG { 21:53:01 DEBUG "status": "success", 21:53:01 DEBUG "data": { 21:53:01 DEBUG "resultType": "vector", 21:53:01 DEBUG "result": [ 21:53:01 DEBUG { 21:53:01 DEBUG "metric": { 21:53:01 DEBUG "instance": "10.142.0.100:9100" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG [...] 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG 21:53:01 DEBUG "instance": "10.142.0.99:9100" 21:53:01 DEBUG }, 21:53:01 DEBUG "value": [ 21:53:01 DEBUG 1686689581, 21:53:01 DEBUG "18804115.53333333" 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG ] 21:53:01 DEBUG } 21:53:01 DEBUG } 21:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689582" 21:53:02 INFO [http_cmd]: http status code OK 21:53:02 DEBUG --- status code --- 21:53:02 DEBUG http status code is 200 (expected 200) 21:53:02 DEBUG --- http response --- 21:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:53:02 DEBUG ----- output ----- 21:53:02 DEBUG { 21:53:02 DEBUG "status": "success", 21:53:02 DEBUG "data": { 21:53:02 DEBUG "resultType": "vector", 21:53:02 DEBUG "result": [ 21:53:02 DEBUG { 21:53:02 DEBUG "metric": { 21:53:02 DEBUG "pod": "ds-idrepo-2" 21:53:02 DEBUG }, 21:53:02 DEBUG "value": [ 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG [...] 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG "pod": "ds-idrepo-0" 21:53:02 DEBUG }, 21:53:02 DEBUG "value": [ 21:53:02 DEBUG 1686689582, 21:53:02 DEBUG "573.2666666666667" 21:53:02 DEBUG ] 21:53:02 DEBUG } 21:53:02 DEBUG ] 21:53:02 DEBUG } 21:53:02 DEBUG } 21:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689582" 21:53:02 INFO [http_cmd]: http status code OK 21:53:02 DEBUG --- status code --- 21:53:02 DEBUG http status code is 200 (expected 200) 21:53:02 DEBUG --- http response --- 21:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:53:02 DEBUG ----- output ----- 21:53:02 DEBUG { 21:53:02 DEBUG "status": "success", 21:53:02 DEBUG "data": { 21:53:02 DEBUG "resultType": "vector", 21:53:02 DEBUG "result": [ 21:53:02 DEBUG { 21:53:02 DEBUG "metric": { 21:53:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:53:02 DEBUG }, 21:53:02 DEBUG "value": [ 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG [...] 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:53:02 DEBUG }, 21:53:02 DEBUG "value": [ 21:53:02 DEBUG 1686689582, 21:53:02 DEBUG "0.23102089125438952" 21:53:02 DEBUG ] 21:53:02 DEBUG } 21:53:02 DEBUG ] 21:53:02 DEBUG } 21:53:02 DEBUG } 21:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689582" 21:53:02 INFO [http_cmd]: http status code OK 21:53:02 DEBUG --- status code --- 21:53:02 DEBUG http status code is 200 (expected 200) 21:53:02 DEBUG --- http response --- 21:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:53:02 DEBUG ----- output ----- 21:53:02 DEBUG { 21:53:02 DEBUG "status": "success", 21:53:02 DEBUG "data": { 21:53:02 DEBUG "resultType": "vector", 21:53:02 DEBUG "result": [ 21:53:02 DEBUG { 21:53:02 DEBUG "metric": { 21:53:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:53:02 DEBUG }, 21:53:02 DEBUG "value": [ 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG [...] 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG 21:53:02 DEBUG "pod": "idm-58dc667486-rgz9j" 21:53:02 DEBUG }, 21:53:02 DEBUG "value": [ 21:53:02 DEBUG 1686689582, 21:53:02 DEBUG "0" 21:53:02 DEBUG ] 21:53:02 DEBUG } 21:53:02 DEBUG ] 21:53:02 DEBUG } 21:53:02 DEBUG } 21:53:17 INFO 21:53:17 INFO [loop_until]: kubectl --namespace=xlou top pods 21:53:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:53:17 INFO [loop_until]: OK (rc = 0) 21:53:17 DEBUG --- stdout --- 21:53:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5699Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3211m 13803Mi ds-idrepo-1 17m 13821Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3093m 4860Mi idm-58dc667486-rgz9j 2481m 5091Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 198m 638Mi 21:53:17 DEBUG --- stderr --- 21:53:17 DEBUG 21:53:20 INFO 21:53:20 INFO [loop_until]: kubectl --namespace=xlou top node 21:53:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:53:20 INFO [loop_until]: OK (rc = 0) 21:53:20 DEBUG --- stdout --- 21:53:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3292m 20% 5804Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6817Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2637m 16% 6075Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 429m 2% 3153Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1194Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13709Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3371m 21% 14519Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 66m 0% 1162Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 78m 0% 14587Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 266m 1% 1955Mi 3% 21:53:20 DEBUG --- stderr --- 21:53:20 DEBUG 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "ds-cts-0" 21:54:01 DEBUG "pod": "ds-cts-2" 21:54:01 DEBUG }, 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "ds-idrepo-0" 21:54:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:54:01 DEBUG }, 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "0" 21:54:01 DEBUG "0" 21:54:01 DEBUG ] 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:54:01 DEBUG "pod": "ds-cts-2" 21:54:01 DEBUG }, 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:54:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:54:01 DEBUG }, 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "0" 21:54:01 DEBUG "0" 21:54:01 DEBUG ] 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "ds-cts-0" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "ds-idrepo-0" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "644.1" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "0.2" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "ds-cts-0" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "ds-cts-2" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "0" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG } 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "ds-idrepo-2" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "ds-idrepo-0" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "644.0666666666666" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "0" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "0" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689641" 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689641" 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "13616435200" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "instance": "10.142.0.100:9100" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "instance": "10.142.0.99:9100" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "0" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "ds-cts-2" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "ds-idrepo-0" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "6845.382380041109" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "0" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689641" 21:54:01 INFO [http_cmd]: http status code OK 21:54:01 DEBUG --- status code --- 21:54:01 DEBUG http status code is 200 (expected 200) 21:54:01 DEBUG --- http response --- 21:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:54:01 DEBUG ----- output ----- 21:54:01 DEBUG { 21:54:01 DEBUG "status": "success", 21:54:01 DEBUG "data": { 21:54:01 DEBUG "resultType": "vector", 21:54:01 DEBUG "result": [ 21:54:01 DEBUG { 21:54:01 DEBUG "metric": { 21:54:01 DEBUG "pod": "ds-idrepo-2" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG [...] 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG 21:54:01 DEBUG "pod": "ds-idrepo-0" 21:54:01 DEBUG }, 21:54:01 DEBUG "value": [ 21:54:01 DEBUG 1686689641, 21:54:01 DEBUG "644.0333333333333" 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG ] 21:54:01 DEBUG } 21:54:01 DEBUG } 21:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689641" 21:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689642" 21:54:02 INFO [http_cmd]: http status code OK 21:54:02 DEBUG --- status code --- 21:54:02 DEBUG http status code is 200 (expected 200) 21:54:02 DEBUG --- http response --- 21:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:54:02 DEBUG ----- output ----- 21:54:02 DEBUG { 21:54:02 DEBUG "status": "success", 21:54:02 DEBUG "data": { 21:54:02 DEBUG "resultType": "vector", 21:54:02 DEBUG "result": [ 21:54:02 DEBUG { 21:54:02 DEBUG "metric": { 21:54:02 DEBUG "instance": "10.142.0.100:9100" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG [...] 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG "instance": "10.142.0.99:9100" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 1686689641, 21:54:02 DEBUG "20611388.766666666" 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG } 21:54:02 INFO [http_cmd]: http status code OK 21:54:02 DEBUG --- status code --- 21:54:02 DEBUG http status code is 200 (expected 200) 21:54:02 DEBUG --- http response --- 21:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:54:02 DEBUG ----- output ----- 21:54:02 DEBUG { 21:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689642" 21:54:02 DEBUG "status": "success", 21:54:02 DEBUG "data": { 21:54:02 DEBUG "resultType": "vector", 21:54:02 DEBUG "result": [ 21:54:02 DEBUG { 21:54:02 DEBUG "metric": { 21:54:02 DEBUG "instance": "10.142.0.100:9100" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG [...] 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG "instance": "10.142.0.99:9100" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 1686689642, 21:54:02 DEBUG "20613544.666666668" 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG } 21:54:02 INFO [http_cmd]: http status code OK 21:54:02 DEBUG --- status code --- 21:54:02 DEBUG http status code is 200 (expected 200) 21:54:02 DEBUG --- http response --- 21:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:54:02 DEBUG ----- output ----- 21:54:02 DEBUG { 21:54:02 DEBUG "status": "success", 21:54:02 DEBUG "data": { 21:54:02 DEBUG "resultType": "vector", 21:54:02 DEBUG "result": [ 21:54:02 DEBUG { 21:54:02 DEBUG "metric": { 21:54:02 DEBUG "instance": "10.142.0.100:9100" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG [...] 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG "instance": "10.142.0.99:9100" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 1686689642, 21:54:02 DEBUG "0.00043333333333445503" 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG } 21:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689642" 21:54:02 INFO [http_cmd]: http status code OK 21:54:02 DEBUG --- status code --- 21:54:02 DEBUG http status code is 200 (expected 200) 21:54:02 DEBUG --- http response --- 21:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:54:02 DEBUG ----- output ----- 21:54:02 DEBUG { 21:54:02 DEBUG "status": "success", 21:54:02 DEBUG "data": { 21:54:02 DEBUG "resultType": "vector", 21:54:02 DEBUG "result": [ 21:54:02 DEBUG { 21:54:02 DEBUG "metric": { 21:54:02 DEBUG "pod": "ds-idrepo-2" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG [...] 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG "pod": "ds-idrepo-0" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 1686689642, 21:54:02 DEBUG "643.9333333333333" 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG } 21:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689642" 21:54:02 INFO [http_cmd]: http status code OK 21:54:02 DEBUG --- status code --- 21:54:02 DEBUG http status code is 200 (expected 200) 21:54:02 DEBUG --- http response --- 21:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:54:02 DEBUG ----- output ----- 21:54:02 DEBUG { 21:54:02 DEBUG "status": "success", 21:54:02 DEBUG "data": { 21:54:02 DEBUG "resultType": "vector", 21:54:02 DEBUG "result": [ 21:54:02 DEBUG { 21:54:02 DEBUG "metric": { 21:54:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG [...] 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 1686689642, 21:54:02 DEBUG "0.24015767932072676" 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG } 21:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689642" 21:54:02 INFO [http_cmd]: http status code OK 21:54:02 DEBUG --- status code --- 21:54:02 DEBUG http status code is 200 (expected 200) 21:54:02 DEBUG --- http response --- 21:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:54:02 DEBUG ----- output ----- 21:54:02 DEBUG { 21:54:02 DEBUG "status": "success", 21:54:02 DEBUG "data": { 21:54:02 DEBUG "resultType": "vector", 21:54:02 DEBUG "result": [ 21:54:02 DEBUG { 21:54:02 DEBUG "metric": { 21:54:02 DEBUG "pod": "ds-cts-2" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG [...] 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG 21:54:02 DEBUG "pod": "ds-idrepo-0" 21:54:02 DEBUG }, 21:54:02 DEBUG "value": [ 21:54:02 DEBUG 1686689642, 21:54:02 DEBUG "0.567013962718832" 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG ] 21:54:02 DEBUG } 21:54:02 DEBUG } 21:54:17 INFO 21:54:17 INFO [loop_until]: kubectl --namespace=xlou top pods 21:54:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:54:17 INFO [loop_until]: OK (rc = 0) 21:54:17 DEBUG --- stdout --- 21:54:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3363m 13800Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3200m 4862Mi idm-58dc667486-rgz9j 2827m 5091Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 210m 639Mi 21:54:17 DEBUG --- stderr --- 21:54:17 DEBUG 21:54:20 INFO 21:54:20 INFO [loop_until]: kubectl --namespace=xlou top node 21:54:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:54:20 INFO [loop_until]: OK (rc = 0) 21:54:20 DEBUG --- stdout --- 21:54:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3249m 20% 5809Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 73m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6817Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2750m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6946Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 439m 2% 3144Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1192Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13708Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3419m 21% 14517Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14590Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 270m 1% 1952Mi 3% 21:54:20 DEBUG --- stderr --- 21:54:20 DEBUG 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG --- status code --- 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- status code --- 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG --- http response --- 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG --- status code --- 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG { 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:01 DEBUG { 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:01 DEBUG "data": { 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG { 21:55:01 DEBUG { 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG { 21:55:01 DEBUG "data": { 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG { 21:55:01 DEBUG "result": [ 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "data": { 21:55:01 DEBUG "data": { 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG { 21:55:01 DEBUG "data": { 21:55:01 DEBUG "result": [ 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "data": { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG { 21:55:01 DEBUG "result": [ 21:55:01 DEBUG "result": [ 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:55:01 DEBUG "result": [ 21:55:01 DEBUG "metric": { 21:55:01 DEBUG { 21:55:01 DEBUG { 21:55:01 DEBUG "result": [ 21:55:01 DEBUG }, 21:55:01 DEBUG { 21:55:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG { 21:55:01 DEBUG "value": [ 21:55:01 DEBUG "metric": { 21:55:01 DEBUG }, 21:55:01 DEBUG "pod": "ds-cts-0" 21:55:01 DEBUG "pod": "ds-cts-0" 21:55:01 DEBUG "metric": { 21:55:01 DEBUG 21:55:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:55:01 DEBUG "value": [ 21:55:01 DEBUG }, 21:55:01 DEBUG }, 21:55:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:55:01 DEBUG 21:55:01 DEBUG }, 21:55:01 DEBUG 21:55:01 DEBUG "value": [ 21:55:01 DEBUG "value": [ 21:55:01 DEBUG }, 21:55:01 DEBUG 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG "value": [ 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG }, 21:55:01 DEBUG 21:55:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 21:55:01 DEBUG }, 21:55:01 DEBUG "pod": "ds-idrepo-0" 21:55:01 DEBUG "pod": "ds-idrepo-0" 21:55:01 DEBUG 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:55:01 DEBUG "value": [ 21:55:01 DEBUG }, 21:55:01 DEBUG }, 21:55:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:55:01 DEBUG "0.2" 21:55:01 DEBUG }, 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG "value": [ 21:55:01 DEBUG }, 21:55:01 DEBUG ] 21:55:01 DEBUG "value": [ 21:55:01 DEBUG "0" 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG } 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG ] 21:55:01 DEBUG "0" 21:55:01 DEBUG "591" 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG ] 21:55:01 DEBUG "205831772.56417337" 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG ] 21:55:01 DEBUG "0.006251851711289062" 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG { 21:55:01 DEBUG { 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "data": { 21:55:01 DEBUG "data": { 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "result": [ 21:55:01 DEBUG "result": [ 21:55:01 DEBUG { 21:55:01 DEBUG { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "pod": "ds-cts-0" 21:55:01 DEBUG "pod": "ds-idrepo-2" 21:55:01 DEBUG }, 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG "pod": "ds-cts-2" 21:55:01 DEBUG "pod": "ds-idrepo-0" 21:55:01 DEBUG }, 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "0" 21:55:01 DEBUG "591" 21:55:01 DEBUG ] 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG { 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "data": { 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "result": [ 21:55:01 DEBUG { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "0" 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG { 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "data": { 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "result": [ 21:55:01 DEBUG { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "0" 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689701" 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689701" 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG { 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "data": { 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "result": [ 21:55:01 DEBUG { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "13617631232" 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG { 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "data": { 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "result": [ 21:55:01 DEBUG { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "instance": "10.142.0.100:9100" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG "instance": "10.142.0.99:9100" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "0.000333333333333267" 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG { 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "data": { 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "result": [ 21:55:01 DEBUG { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "0" 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 INFO [http_cmd]: http status code OK 21:55:01 DEBUG --- status code --- 21:55:01 DEBUG http status code is 200 (expected 200) 21:55:01 DEBUG --- http response --- 21:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:01 DEBUG ----- output ----- 21:55:01 DEBUG { 21:55:01 DEBUG "status": "success", 21:55:01 DEBUG "data": { 21:55:01 DEBUG "resultType": "vector", 21:55:01 DEBUG "result": [ 21:55:01 DEBUG { 21:55:01 DEBUG "metric": { 21:55:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG [...] 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG 21:55:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:55:01 DEBUG }, 21:55:01 DEBUG "value": [ 21:55:01 DEBUG 1686689701, 21:55:01 DEBUG "0" 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG ] 21:55:01 DEBUG } 21:55:01 DEBUG } 21:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689701" 21:55:02 INFO [http_cmd]: http status code OK 21:55:02 DEBUG --- status code --- 21:55:02 DEBUG http status code is 200 (expected 200) 21:55:02 DEBUG --- http response --- 21:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:55:02 DEBUG ----- output ----- 21:55:02 DEBUG { 21:55:02 DEBUG "status": "success", 21:55:02 DEBUG "data": { 21:55:02 DEBUG "resultType": "vector", 21:55:02 DEBUG "result": [ 21:55:02 DEBUG { 21:55:02 DEBUG "metric": { 21:55:02 DEBUG "pod": "ds-idrepo-2" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG [...] 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG "pod": "ds-idrepo-0" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 1686689701, 21:55:02 DEBUG "591.0333333333333" 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG } 21:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689702" 21:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689702" 21:55:02 INFO [http_cmd]: http status code OK 21:55:02 DEBUG --- status code --- 21:55:02 DEBUG http status code is 200 (expected 200) 21:55:02 DEBUG --- http response --- 21:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:02 DEBUG ----- output ----- 21:55:02 DEBUG { 21:55:02 DEBUG "status": "success", 21:55:02 DEBUG "data": { 21:55:02 DEBUG "resultType": "vector", 21:55:02 DEBUG "result": [ 21:55:02 DEBUG { 21:55:02 DEBUG "metric": { 21:55:02 DEBUG "instance": "10.142.0.100:9100" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG [...] 21:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689702" 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG "instance": "10.142.0.99:9100" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 1686689702, 21:55:02 DEBUG "19716125.933333334" 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG } 21:55:02 INFO [http_cmd]: http status code OK 21:55:02 DEBUG --- status code --- 21:55:02 DEBUG http status code is 200 (expected 200) 21:55:02 DEBUG --- http response --- 21:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:02 DEBUG ----- output ----- 21:55:02 DEBUG { 21:55:02 DEBUG "status": "success", 21:55:02 DEBUG "data": { 21:55:02 DEBUG "resultType": "vector", 21:55:02 DEBUG "result": [ 21:55:02 DEBUG { 21:55:02 DEBUG "metric": { 21:55:02 DEBUG "instance": "10.142.0.100:9100" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG [...] 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG "instance": "10.142.0.99:9100" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 1686689702, 21:55:02 DEBUG "19718192.5" 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG } 21:55:02 INFO [http_cmd]: http status code OK 21:55:02 DEBUG --- status code --- 21:55:02 DEBUG http status code is 200 (expected 200) 21:55:02 DEBUG --- http response --- 21:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:02 DEBUG ----- output ----- 21:55:02 DEBUG { 21:55:02 DEBUG "status": "success", 21:55:02 DEBUG "data": { 21:55:02 DEBUG "resultType": "vector", 21:55:02 DEBUG "result": [ 21:55:02 DEBUG { 21:55:02 DEBUG "metric": { 21:55:02 DEBUG "instance": "10.142.0.100:9100" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG [...] 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG "instance": "10.142.0.99:9100" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 1686689702, 21:55:02 DEBUG "0.00046666666666700016" 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG } 21:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689702" 21:55:02 INFO [http_cmd]: http status code OK 21:55:02 DEBUG --- status code --- 21:55:02 DEBUG http status code is 200 (expected 200) 21:55:02 DEBUG --- http response --- 21:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:55:02 DEBUG ----- output ----- 21:55:02 DEBUG { 21:55:02 DEBUG "status": "success", 21:55:02 DEBUG "data": { 21:55:02 DEBUG "resultType": "vector", 21:55:02 DEBUG "result": [ 21:55:02 DEBUG { 21:55:02 DEBUG "metric": { 21:55:02 DEBUG "pod": "ds-idrepo-2" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG [...] 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG "pod": "ds-idrepo-0" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 1686689702, 21:55:02 DEBUG "591.0333333333333" 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG } 21:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689702" 21:55:02 INFO [http_cmd]: http status code OK 21:55:02 DEBUG --- status code --- 21:55:02 DEBUG http status code is 200 (expected 200) 21:55:02 DEBUG --- http response --- 21:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:02 DEBUG ----- output ----- 21:55:02 DEBUG { 21:55:02 DEBUG "status": "success", 21:55:02 DEBUG "data": { 21:55:02 DEBUG "resultType": "vector", 21:55:02 DEBUG "result": [ 21:55:02 DEBUG { 21:55:02 DEBUG "metric": { 21:55:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG [...] 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 1686689702, 21:55:02 DEBUG "0.22298418091479574" 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG } 21:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689702" 21:55:02 INFO [http_cmd]: http status code OK 21:55:02 DEBUG --- status code --- 21:55:02 DEBUG http status code is 200 (expected 200) 21:55:02 DEBUG --- http response --- 21:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:55:02 DEBUG ----- output ----- 21:55:02 DEBUG { 21:55:02 DEBUG "status": "success", 21:55:02 DEBUG "data": { 21:55:02 DEBUG "resultType": "vector", 21:55:02 DEBUG "result": [ 21:55:02 DEBUG { 21:55:02 DEBUG "metric": { 21:55:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG [...] 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG 21:55:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:55:02 DEBUG }, 21:55:02 DEBUG "value": [ 21:55:02 DEBUG 1686689702, 21:55:02 DEBUG "0" 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG ] 21:55:02 DEBUG } 21:55:02 DEBUG } 21:55:17 INFO 21:55:17 INFO [loop_until]: kubectl --namespace=xlou top pods 21:55:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:55:17 INFO [loop_until]: OK (rc = 0) 21:55:17 DEBUG --- stdout --- 21:55:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 10m 416Mi ds-cts-1 8m 422Mi ds-cts-2 8m 392Mi ds-idrepo-0 3290m 13801Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2959m 4864Mi idm-58dc667486-rgz9j 2981m 5091Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 206m 639Mi 21:55:17 DEBUG --- stderr --- 21:55:17 DEBUG 21:55:20 INFO 21:55:20 INFO [loop_until]: kubectl --namespace=xlou top node 21:55:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:55:20 INFO [loop_until]: OK (rc = 0) 21:55:20 DEBUG --- stdout --- 21:55:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3059m 19% 5810Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 78m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2748m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6952Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 447m 2% 3168Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13710Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3377m 21% 14501Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 66m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14591Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 266m 1% 1952Mi 3% 21:55:20 DEBUG --- stderr --- 21:55:20 DEBUG 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "ds-cts-0" 21:56:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:56:01 DEBUG }, 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "ds-idrepo-0" 21:56:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:56:01 DEBUG }, 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "622.8" 21:56:01 DEBUG "0.2" 21:56:01 DEBUG ] 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG ] 21:56:01 DEBUG ] 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "ds-cts-0" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "ds-idrepo-0" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "ds-idrepo-2" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "ds-idrepo-0" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "622.8666666666667" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "ds-cts-0" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "ds-cts-2" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689761" 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689761" 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "13617111040" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "instance": "10.142.0.100:9100" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "instance": "10.142.0.99:9100" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689761" 21:56:01 INFO [http_cmd]: http status code OK 21:56:01 DEBUG --- status code --- 21:56:01 DEBUG http status code is 200 (expected 200) 21:56:01 DEBUG --- http response --- 21:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:56:01 DEBUG ----- output ----- 21:56:01 DEBUG { 21:56:01 DEBUG "status": "success", 21:56:01 DEBUG "data": { 21:56:01 DEBUG "resultType": "vector", 21:56:01 DEBUG "result": [ 21:56:01 DEBUG { 21:56:01 DEBUG "metric": { 21:56:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG [...] 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG 21:56:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:56:01 DEBUG }, 21:56:01 DEBUG "value": [ 21:56:01 DEBUG 1686689761, 21:56:01 DEBUG "0" 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG ] 21:56:01 DEBUG } 21:56:01 DEBUG } 21:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689762" 21:56:02 INFO [http_cmd]: http status code OK 21:56:02 DEBUG --- status code --- 21:56:02 DEBUG http status code is 200 (expected 200) 21:56:02 DEBUG --- http response --- 21:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:56:02 DEBUG ----- output ----- 21:56:02 DEBUG { 21:56:02 DEBUG "status": "success", 21:56:02 DEBUG "data": { 21:56:02 DEBUG "resultType": "vector", 21:56:02 DEBUG "result": [ 21:56:02 DEBUG { 21:56:02 DEBUG "metric": { 21:56:02 DEBUG "pod": "ds-idrepo-2" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG [...] 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG "pod": "ds-idrepo-0" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 1686689762, 21:56:02 DEBUG "622.9333333333333" 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG } 21:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689762" 21:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689762" 21:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689762" 21:56:02 INFO [http_cmd]: http status code OK 21:56:02 DEBUG --- status code --- 21:56:02 DEBUG http status code is 200 (expected 200) 21:56:02 DEBUG --- http response --- 21:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:56:02 DEBUG ----- output ----- 21:56:02 DEBUG { 21:56:02 DEBUG "status": "success", 21:56:02 DEBUG "data": { 21:56:02 DEBUG "resultType": "vector", 21:56:02 DEBUG "result": [ 21:56:02 DEBUG { 21:56:02 DEBUG "metric": { 21:56:02 DEBUG "instance": "10.142.0.100:9100" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG [...] 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG "instance": "10.142.0.99:9100" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 1686689762, 21:56:02 DEBUG "19941884.233333334" 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG } 21:56:02 INFO [http_cmd]: http status code OK 21:56:02 DEBUG --- status code --- 21:56:02 DEBUG http status code is 200 (expected 200) 21:56:02 DEBUG --- http response --- 21:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:56:02 DEBUG ----- output ----- 21:56:02 DEBUG { 21:56:02 DEBUG "status": "success", 21:56:02 DEBUG "data": { 21:56:02 DEBUG "resultType": "vector", 21:56:02 DEBUG "result": [ 21:56:02 DEBUG { 21:56:02 DEBUG "metric": { 21:56:02 DEBUG "instance": "10.142.0.100:9100" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG [...] 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG "instance": "10.142.0.99:9100" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 1686689762, 21:56:02 DEBUG "19941756.966666665" 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG } 21:56:02 INFO [http_cmd]: http status code OK 21:56:02 DEBUG --- status code --- 21:56:02 DEBUG http status code is 200 (expected 200) 21:56:02 DEBUG --- http response --- 21:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:56:02 DEBUG ----- output ----- 21:56:02 DEBUG { 21:56:02 DEBUG "status": "success", 21:56:02 DEBUG "data": { 21:56:02 DEBUG "resultType": "vector", 21:56:02 DEBUG "result": [ 21:56:02 DEBUG { 21:56:02 DEBUG "metric": { 21:56:02 DEBUG "instance": "10.142.0.100:9100" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG [...] 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG "instance": "10.142.0.99:9100" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 1686689762, 21:56:02 DEBUG "0.0004333333333325603" 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG } 21:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689762" 21:56:02 INFO [http_cmd]: http status code OK 21:56:02 DEBUG --- status code --- 21:56:02 DEBUG http status code is 200 (expected 200) 21:56:02 DEBUG --- http response --- 21:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:56:02 DEBUG ----- output ----- 21:56:02 DEBUG { 21:56:02 DEBUG "status": "success", 21:56:02 DEBUG "data": { 21:56:02 DEBUG "resultType": "vector", 21:56:02 DEBUG "result": [ 21:56:02 DEBUG { 21:56:02 DEBUG "metric": { 21:56:02 DEBUG "pod": "ds-idrepo-2" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG [...] 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG "pod": "ds-idrepo-0" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 1686689762, 21:56:02 DEBUG "622.9666666666667" 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG } 21:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689762" 21:56:02 INFO [http_cmd]: http status code OK 21:56:02 DEBUG --- status code --- 21:56:02 DEBUG http status code is 200 (expected 200) 21:56:02 DEBUG --- http response --- 21:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:56:02 DEBUG ----- output ----- 21:56:02 DEBUG { 21:56:02 DEBUG "status": "success", 21:56:02 DEBUG "data": { 21:56:02 DEBUG "resultType": "vector", 21:56:02 DEBUG "result": [ 21:56:02 DEBUG { 21:56:02 DEBUG "metric": { 21:56:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG [...] 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 1686689762, 21:56:02 DEBUG "0.22447346108748895" 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG } 21:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689762" 21:56:02 INFO [http_cmd]: http status code OK 21:56:02 DEBUG --- status code --- 21:56:02 DEBUG http status code is 200 (expected 200) 21:56:02 DEBUG --- http response --- 21:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:56:02 DEBUG ----- output ----- 21:56:02 DEBUG { 21:56:02 DEBUG "status": "success", 21:56:02 DEBUG "data": { 21:56:02 DEBUG "resultType": "vector", 21:56:02 DEBUG "result": [ 21:56:02 DEBUG { 21:56:02 DEBUG "metric": { 21:56:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG [...] 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG 21:56:02 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 21:56:02 DEBUG }, 21:56:02 DEBUG "value": [ 21:56:02 DEBUG 1686689762, 21:56:02 DEBUG "0" 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG ] 21:56:02 DEBUG } 21:56:02 DEBUG } 21:56:17 INFO 21:56:17 INFO [loop_until]: kubectl --namespace=xlou top pods 21:56:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:56:17 INFO [loop_until]: OK (rc = 0) 21:56:17 DEBUG --- stdout --- 21:56:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 423Mi ds-cts-2 9m 392Mi ds-idrepo-0 3358m 13805Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3311m 4866Mi idm-58dc667486-rgz9j 2598m 5092Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 209m 639Mi 21:56:17 DEBUG --- stderr --- 21:56:17 DEBUG 21:56:20 INFO 21:56:20 INFO [loop_until]: kubectl --namespace=xlou top node 21:56:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:56:21 INFO [loop_until]: OK (rc = 0) 21:56:21 DEBUG --- stdout --- 21:56:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3390m 21% 5813Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6817Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2807m 17% 6089Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 68m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 435m 2% 3165Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 85m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 63m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1191Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13712Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3591m 22% 14506Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14590Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 288m 1% 1950Mi 3% 21:56:21 DEBUG --- stderr --- 21:56:21 DEBUG 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG --- status code --- 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- status code --- 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:57:01 DEBUG { 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG { 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG { 21:57:01 DEBUG { 21:57:01 DEBUG { 21:57:01 DEBUG "data": { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG { 21:57:01 DEBUG { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "data": { 21:57:01 DEBUG "data": { 21:57:01 DEBUG "result": [ 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "data": { 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG { 21:57:01 DEBUG "result": [ 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "result": [ 21:57:01 DEBUG "result": [ 21:57:01 DEBUG "result": [ 21:57:01 DEBUG "metric": { 21:57:01 DEBUG { 21:57:01 DEBUG "result": [ 21:57:01 DEBUG "result": [ 21:57:01 DEBUG { 21:57:01 DEBUG { 21:57:01 DEBUG { 21:57:01 DEBUG "pod": "ds-cts-0" 21:57:01 DEBUG "metric": { 21:57:01 DEBUG { 21:57:01 DEBUG { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG }, 21:57:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "pod": "ds-cts-0" 21:57:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:57:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:57:01 DEBUG "value": [ 21:57:01 DEBUG }, 21:57:01 DEBUG "pod": "ds-idrepo-2" 21:57:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:57:01 DEBUG }, 21:57:01 DEBUG }, 21:57:01 DEBUG }, 21:57:01 DEBUG 21:57:01 DEBUG "value": [ 21:57:01 DEBUG }, 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG "value": [ 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "value": [ 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG [...] 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "pod": "ds-idrepo-0" 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG }, 21:57:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "pod": "ds-idrepo-0" 21:57:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:57:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:57:01 DEBUG "value": [ 21:57:01 DEBUG }, 21:57:01 DEBUG "pod": "ds-idrepo-0" 21:57:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:57:01 DEBUG }, 21:57:01 DEBUG }, 21:57:01 DEBUG }, 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG }, 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG "value": [ 21:57:01 DEBUG "value": [ 21:57:01 DEBUG "633.6" 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG ] 21:57:01 DEBUG "0.2" 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "0" 21:57:01 DEBUG "2.744161531352877" 21:57:01 DEBUG "0" 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG "633.5666666666666" 21:57:01 DEBUG "20329.049224045335" 21:57:01 DEBUG ] 21:57:01 DEBUG ] 21:57:01 DEBUG ] 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG ] 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "result": [ 21:57:01 DEBUG { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "0" 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "result": [ 21:57:01 DEBUG { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "pod": "ds-cts-0" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "pod": "ds-cts-2" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "0" 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689821" 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "result": [ 21:57:01 DEBUG { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "0" 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689821" 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "result": [ 21:57:01 DEBUG { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "instance": "10.142.0.100:9100" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "instance": "10.142.0.99:9100" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "0" 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "result": [ 21:57:01 DEBUG { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "13619499008" 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "result": [ 21:57:01 DEBUG { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "pod": "idm-58dc667486-rgz9j" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "0" 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689821" 21:57:01 INFO [http_cmd]: http status code OK 21:57:01 DEBUG --- status code --- 21:57:01 DEBUG http status code is 200 (expected 200) 21:57:01 DEBUG --- http response --- 21:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:57:01 DEBUG ----- output ----- 21:57:01 DEBUG { 21:57:01 DEBUG "status": "success", 21:57:01 DEBUG "data": { 21:57:01 DEBUG "resultType": "vector", 21:57:01 DEBUG "result": [ 21:57:01 DEBUG { 21:57:01 DEBUG "metric": { 21:57:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG [...] 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG 21:57:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:57:01 DEBUG }, 21:57:01 DEBUG "value": [ 21:57:01 DEBUG 1686689821, 21:57:01 DEBUG "0" 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG ] 21:57:01 DEBUG } 21:57:01 DEBUG } 21:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689822" 21:57:02 INFO [http_cmd]: http status code OK 21:57:02 DEBUG --- status code --- 21:57:02 DEBUG http status code is 200 (expected 200) 21:57:02 DEBUG --- http response --- 21:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:57:02 DEBUG ----- output ----- 21:57:02 DEBUG { 21:57:02 DEBUG "status": "success", 21:57:02 DEBUG "data": { 21:57:02 DEBUG "resultType": "vector", 21:57:02 DEBUG "result": [ 21:57:02 DEBUG { 21:57:02 DEBUG "metric": { 21:57:02 DEBUG "pod": "ds-idrepo-2" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG [...] 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG "pod": "ds-idrepo-0" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 1686689822, 21:57:02 DEBUG "633.5333333333333" 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG } 21:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689822" 21:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689822" 21:57:02 INFO [http_cmd]: http status code OK 21:57:02 DEBUG --- status code --- 21:57:02 DEBUG http status code is 200 (expected 200) 21:57:02 DEBUG --- http response --- 21:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:57:02 DEBUG ----- output ----- 21:57:02 DEBUG { 21:57:02 DEBUG "status": "success", 21:57:02 DEBUG "data": { 21:57:02 DEBUG "resultType": "vector", 21:57:02 DEBUG "result": [ 21:57:02 DEBUG { 21:57:02 DEBUG "metric": { 21:57:02 DEBUG "instance": "10.142.0.100:9100" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG [...] 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG "instance": "10.142.0.99:9100" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 1686689822, 21:57:02 DEBUG "20876205.933333334" 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG } 21:57:02 INFO [http_cmd]: http status code OK 21:57:02 DEBUG --- status code --- 21:57:02 DEBUG http status code is 200 (expected 200) 21:57:02 DEBUG --- http response --- 21:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:57:02 DEBUG ----- output ----- 21:57:02 DEBUG { 21:57:02 DEBUG "status": "success", 21:57:02 DEBUG "data": { 21:57:02 DEBUG "resultType": "vector", 21:57:02 DEBUG "result": [ 21:57:02 DEBUG { 21:57:02 DEBUG "metric": { 21:57:02 DEBUG "instance": "10.142.0.100:9100" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG [...] 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG "instance": "10.142.0.99:9100" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 1686689822, 21:57:02 DEBUG "20874201.799999997" 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG } 21:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689822" 21:57:02 INFO [http_cmd]: http status code OK 21:57:02 DEBUG --- status code --- 21:57:02 DEBUG http status code is 200 (expected 200) 21:57:02 DEBUG --- http response --- 21:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:57:02 DEBUG ----- output ----- 21:57:02 DEBUG { 21:57:02 DEBUG "status": "success", 21:57:02 DEBUG "data": { 21:57:02 DEBUG "resultType": "vector", 21:57:02 DEBUG "result": [ 21:57:02 DEBUG { 21:57:02 DEBUG "metric": { 21:57:02 DEBUG "instance": "10.142.0.100:9100" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG [...] 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG "instance": "10.142.0.99:9100" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 1686689822, 21:57:02 DEBUG "0.00026666666666604516" 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG } 21:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689822" 21:57:02 INFO [http_cmd]: http status code OK 21:57:02 DEBUG --- status code --- 21:57:02 DEBUG http status code is 200 (expected 200) 21:57:02 DEBUG --- http response --- 21:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:57:02 DEBUG ----- output ----- 21:57:02 DEBUG { 21:57:02 DEBUG "status": "success", 21:57:02 DEBUG "data": { 21:57:02 DEBUG "resultType": "vector", 21:57:02 DEBUG "result": [ 21:57:02 DEBUG { 21:57:02 DEBUG "metric": { 21:57:02 DEBUG "pod": "ds-idrepo-2" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG [...] 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG "pod": "ds-idrepo-0" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 1686689822, 21:57:02 DEBUG "633.5333333333333" 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG } 21:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689822" 21:57:02 INFO [http_cmd]: http status code OK 21:57:02 DEBUG --- status code --- 21:57:02 DEBUG http status code is 200 (expected 200) 21:57:02 DEBUG --- http response --- 21:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:57:02 DEBUG ----- output ----- 21:57:02 DEBUG { 21:57:02 DEBUG "status": "success", 21:57:02 DEBUG "data": { 21:57:02 DEBUG "resultType": "vector", 21:57:02 DEBUG "result": [ 21:57:02 DEBUG { 21:57:02 DEBUG "metric": { 21:57:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG [...] 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 1686689822, 21:57:02 DEBUG "0.24213739181137198" 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG } 21:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689822" 21:57:02 INFO [http_cmd]: http status code OK 21:57:02 DEBUG --- status code --- 21:57:02 DEBUG http status code is 200 (expected 200) 21:57:02 DEBUG --- http response --- 21:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:57:02 DEBUG ----- output ----- 21:57:02 DEBUG { 21:57:02 DEBUG "status": "success", 21:57:02 DEBUG "data": { 21:57:02 DEBUG "resultType": "vector", 21:57:02 DEBUG "result": [ 21:57:02 DEBUG { 21:57:02 DEBUG "metric": { 21:57:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG [...] 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG 21:57:02 DEBUG "pod": "idm-58dc667486-rgz9j" 21:57:02 DEBUG }, 21:57:02 DEBUG "value": [ 21:57:02 DEBUG 1686689822, 21:57:02 DEBUG "0" 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG ] 21:57:02 DEBUG } 21:57:02 DEBUG } 21:57:17 INFO 21:57:17 INFO [loop_until]: kubectl --namespace=xlou top pods 21:57:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:57:17 INFO [loop_until]: OK (rc = 0) 21:57:17 DEBUG --- stdout --- 21:57:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 6m 5698Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 10m 416Mi ds-cts-1 9m 423Mi ds-cts-2 8m 392Mi ds-idrepo-0 3442m 13802Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2913m 4868Mi idm-58dc667486-rgz9j 2890m 5092Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 205m 639Mi 21:57:17 DEBUG --- stderr --- 21:57:17 DEBUG 21:57:21 INFO 21:57:21 INFO [loop_until]: kubectl --namespace=xlou top node 21:57:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:57:21 INFO [loop_until]: OK (rc = 0) 21:57:21 DEBUG --- stdout --- 21:57:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3159m 19% 5816Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 68m 0% 6816Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2981m 18% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 69m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 441m 2% 3167Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1171Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1193Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13711Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3454m 21% 14500Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14590Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 263m 1% 1952Mi 3% 21:57:21 DEBUG --- stderr --- 21:57:21 DEBUG 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG { 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG "data": { 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "result": [ 21:58:01 DEBUG { 21:58:01 DEBUG "metric": { 21:58:01 DEBUG "pod": "ds-idrepo-2" 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG [...] 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG "pod": "ds-idrepo-0" 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG "619.6666666666666" 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG { 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG "data": { 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "result": [ 21:58:01 DEBUG { 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 DEBUG "metric": { 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:58:01 DEBUG { 21:58:01 DEBUG }, 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG "value": [ 21:58:01 DEBUG "data": { 21:58:01 DEBUG 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG 21:58:01 DEBUG "result": [ 21:58:01 DEBUG 21:58:01 DEBUG { 21:58:01 DEBUG [...] 21:58:01 DEBUG "metric": { 21:58:01 DEBUG 21:58:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:58:01 DEBUG 21:58:01 DEBUG }, 21:58:01 DEBUG 21:58:01 DEBUG "value": [ 21:58:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:58:01 DEBUG 21:58:01 DEBUG }, 21:58:01 DEBUG 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG [...] 21:58:01 DEBUG "0" 21:58:01 DEBUG 21:58:01 DEBUG ] 21:58:01 DEBUG 21:58:01 DEBUG } 21:58:01 DEBUG 21:58:01 DEBUG ] 21:58:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:58:01 DEBUG } 21:58:01 DEBUG }, 21:58:01 DEBUG } 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG "0.006217971049865007" 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG --- status code --- 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG --- status code --- 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG { 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:58:01 DEBUG "data": { 21:58:01 DEBUG { 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG { 21:58:01 DEBUG "result": [ 21:58:01 DEBUG "data": { 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG { 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "data": { 21:58:01 DEBUG "metric": { 21:58:01 DEBUG "result": [ 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:58:01 DEBUG { 21:58:01 DEBUG "result": [ 21:58:01 DEBUG }, 21:58:01 DEBUG "metric": { 21:58:01 DEBUG { 21:58:01 DEBUG "value": [ 21:58:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:58:01 DEBUG "metric": { 21:58:01 DEBUG 21:58:01 DEBUG }, 21:58:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:58:01 DEBUG 21:58:01 DEBUG "value": [ 21:58:01 DEBUG }, 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG "value": [ 21:58:01 DEBUG [...] 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG [...] 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG [...] 21:58:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG }, 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG "value": [ 21:58:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:58:01 DEBUG 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG }, 21:58:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:58:01 DEBUG "0.2" 21:58:01 DEBUG "value": [ 21:58:01 DEBUG }, 21:58:01 DEBUG ] 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG } 21:58:01 DEBUG "0" 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG ] 21:58:01 DEBUG ] 21:58:01 DEBUG "8886.538526729719" 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG { 21:58:01 DEBUG { 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG "data": { 21:58:01 DEBUG "data": { 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "result": [ 21:58:01 DEBUG "result": [ 21:58:01 DEBUG { 21:58:01 DEBUG { 21:58:01 DEBUG "metric": { 21:58:01 DEBUG "metric": { 21:58:01 DEBUG "pod": "ds-cts-0" 21:58:01 DEBUG "pod": "ds-cts-0" 21:58:01 DEBUG }, 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG [...] 21:58:01 DEBUG [...] 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG "pod": "ds-idrepo-0" 21:58:01 DEBUG "pod": "ds-idrepo-0" 21:58:01 DEBUG }, 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG "0" 21:58:01 DEBUG "619.6666666666666" 21:58:01 DEBUG ] 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG { 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG "data": { 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "result": [ 21:58:01 DEBUG { 21:58:01 DEBUG "metric": { 21:58:01 DEBUG "pod": "ds-cts-0" 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG [...] 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG "pod": "ds-cts-2" 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG "0" 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG { 21:58:01 DEBUG "status": "success", 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689881" 21:58:01 DEBUG "data": { 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "result": [ 21:58:01 DEBUG { 21:58:01 DEBUG "metric": { 21:58:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG [...] 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG "0" 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689881" 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG { 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG "data": { 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG "result": [ 21:58:01 DEBUG { 21:58:01 DEBUG { 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG "metric": { 21:58:01 DEBUG "data": { 21:58:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG }, 21:58:01 DEBUG "result": [ 21:58:01 DEBUG "value": [ 21:58:01 DEBUG { 21:58:01 DEBUG 21:58:01 DEBUG "metric": { 21:58:01 DEBUG 21:58:01 DEBUG "instance": "10.142.0.100:9100" 21:58:01 DEBUG 21:58:01 DEBUG }, 21:58:01 DEBUG [...] 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:58:01 DEBUG [...] 21:58:01 DEBUG }, 21:58:01 DEBUG 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG 21:58:01 DEBUG "13616574464" 21:58:01 DEBUG "instance": "10.142.0.99:9100" 21:58:01 DEBUG ] 21:58:01 DEBUG }, 21:58:01 DEBUG } 21:58:01 DEBUG "value": [ 21:58:01 DEBUG ] 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG } 21:58:01 DEBUG "0.0003333333333333262" 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689881" 21:58:01 INFO [http_cmd]: http status code OK 21:58:01 DEBUG --- status code --- 21:58:01 DEBUG http status code is 200 (expected 200) 21:58:01 DEBUG --- http response --- 21:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:58:01 DEBUG ----- output ----- 21:58:01 DEBUG { 21:58:01 DEBUG "status": "success", 21:58:01 DEBUG "data": { 21:58:01 DEBUG "resultType": "vector", 21:58:01 DEBUG "result": [ 21:58:01 DEBUG { 21:58:01 DEBUG "metric": { 21:58:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG [...] 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG 21:58:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:58:01 DEBUG }, 21:58:01 DEBUG "value": [ 21:58:01 DEBUG 1686689881, 21:58:01 DEBUG "0" 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG ] 21:58:01 DEBUG } 21:58:01 DEBUG } 21:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689882" 21:58:02 INFO [http_cmd]: http status code OK 21:58:02 DEBUG --- status code --- 21:58:02 DEBUG http status code is 200 (expected 200) 21:58:02 DEBUG --- http response --- 21:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:58:02 DEBUG ----- output ----- 21:58:02 DEBUG { 21:58:02 DEBUG "status": "success", 21:58:02 DEBUG "data": { 21:58:02 DEBUG "resultType": "vector", 21:58:02 DEBUG "result": [ 21:58:02 DEBUG { 21:58:02 DEBUG "metric": { 21:58:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG [...] 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 1686689882, 21:58:02 DEBUG "0" 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG } 21:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689882" 21:58:02 INFO [http_cmd]: http status code OK 21:58:02 DEBUG --- status code --- 21:58:02 DEBUG http status code is 200 (expected 200) 21:58:02 DEBUG --- http response --- 21:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:58:02 DEBUG ----- output ----- 21:58:02 DEBUG { 21:58:02 DEBUG "status": "success", 21:58:02 DEBUG "data": { 21:58:02 DEBUG "resultType": "vector", 21:58:02 DEBUG "result": [ 21:58:02 DEBUG { 21:58:02 DEBUG "metric": { 21:58:02 DEBUG "pod": "ds-idrepo-2" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG [...] 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG "pod": "ds-idrepo-0" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 1686689882, 21:58:02 DEBUG "619.6666666666666" 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG } 21:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689882" 21:58:02 INFO [http_cmd]: http status code OK 21:58:02 DEBUG --- status code --- 21:58:02 DEBUG http status code is 200 (expected 200) 21:58:02 DEBUG --- http response --- 21:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:58:02 DEBUG ----- output ----- 21:58:02 DEBUG { 21:58:02 DEBUG "status": "success", 21:58:02 DEBUG "data": { 21:58:02 DEBUG "resultType": "vector", 21:58:02 DEBUG "result": [ 21:58:02 DEBUG { 21:58:02 DEBUG "metric": { 21:58:02 DEBUG "instance": "10.142.0.100:9100" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG [...] 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG "instance": "10.142.0.99:9100" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 1686689882, 21:58:02 DEBUG "21376702.333333336" 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG } 21:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689882" 21:58:02 INFO [http_cmd]: http status code OK 21:58:02 DEBUG --- status code --- 21:58:02 DEBUG http status code is 200 (expected 200) 21:58:02 DEBUG --- http response --- 21:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:58:02 DEBUG ----- output ----- 21:58:02 DEBUG { 21:58:02 DEBUG "status": "success", 21:58:02 DEBUG "data": { 21:58:02 DEBUG "resultType": "vector", 21:58:02 DEBUG "result": [ 21:58:02 DEBUG { 21:58:02 DEBUG "metric": { 21:58:02 DEBUG "instance": "10.142.0.100:9100" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG [...] 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG "instance": "10.142.0.99:9100" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 1686689882, 21:58:02 DEBUG "21376489.166666664" 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG } 21:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689882" 21:58:02 INFO [http_cmd]: http status code OK 21:58:02 DEBUG --- status code --- 21:58:02 DEBUG http status code is 200 (expected 200) 21:58:02 DEBUG --- http response --- 21:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:58:02 DEBUG ----- output ----- 21:58:02 DEBUG { 21:58:02 DEBUG "status": "success", 21:58:02 DEBUG "data": { 21:58:02 DEBUG "resultType": "vector", 21:58:02 DEBUG "result": [ 21:58:02 DEBUG { 21:58:02 DEBUG "metric": { 21:58:02 DEBUG "instance": "10.142.0.100:9100" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG [...] 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG "instance": "10.142.0.99:9100" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 1686689882, 21:58:02 DEBUG "0.00023333333333350008" 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG } 21:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689882" 21:58:02 INFO [http_cmd]: http status code OK 21:58:02 DEBUG --- status code --- 21:58:02 DEBUG http status code is 200 (expected 200) 21:58:02 DEBUG --- http response --- 21:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:58:02 DEBUG ----- output ----- 21:58:02 DEBUG { 21:58:02 DEBUG "status": "success", 21:58:02 DEBUG "data": { 21:58:02 DEBUG "resultType": "vector", 21:58:02 DEBUG "result": [ 21:58:02 DEBUG { 21:58:02 DEBUG "metric": { 21:58:02 DEBUG "pod": "ds-idrepo-2" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG [...] 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG "pod": "ds-idrepo-0" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 1686689882, 21:58:02 DEBUG "619.7" 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG } 21:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689882" 21:58:02 INFO [http_cmd]: http status code OK 21:58:02 DEBUG --- status code --- 21:58:02 DEBUG http status code is 200 (expected 200) 21:58:02 DEBUG --- http response --- 21:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:58:02 DEBUG ----- output ----- 21:58:02 DEBUG { 21:58:02 DEBUG "status": "success", 21:58:02 DEBUG "data": { 21:58:02 DEBUG "resultType": "vector", 21:58:02 DEBUG "result": [ 21:58:02 DEBUG { 21:58:02 DEBUG "metric": { 21:58:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG [...] 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 1686689882, 21:58:02 DEBUG "0.22877338937609307" 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG } 21:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689882" 21:58:02 INFO [http_cmd]: http status code OK 21:58:02 DEBUG --- status code --- 21:58:02 DEBUG http status code is 200 (expected 200) 21:58:02 DEBUG --- http response --- 21:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:58:02 DEBUG ----- output ----- 21:58:02 DEBUG { 21:58:02 DEBUG "status": "success", 21:58:02 DEBUG "data": { 21:58:02 DEBUG "resultType": "vector", 21:58:02 DEBUG "result": [ 21:58:02 DEBUG { 21:58:02 DEBUG "metric": { 21:58:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG [...] 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG 21:58:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:58:02 DEBUG }, 21:58:02 DEBUG "value": [ 21:58:02 DEBUG 1686689882, 21:58:02 DEBUG "0" 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG ] 21:58:02 DEBUG } 21:58:02 DEBUG } 21:58:17 INFO 21:58:17 INFO [loop_until]: kubectl --namespace=xlou top pods 21:58:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:58:17 INFO [loop_until]: OK (rc = 0) 21:58:17 DEBUG --- stdout --- 21:58:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 10m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 10m 425Mi ds-cts-2 9m 392Mi ds-idrepo-0 3267m 13805Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2951m 4869Mi idm-58dc667486-rgz9j 2895m 5092Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 200m 639Mi 21:58:17 DEBUG --- stderr --- 21:58:17 DEBUG 21:58:21 INFO 21:58:21 INFO [loop_until]: kubectl --namespace=xlou top node 21:58:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:58:21 INFO [loop_until]: OK (rc = 0) 21:58:21 DEBUG --- stdout --- 21:58:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2969m 18% 5817Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6815Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2717m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 429m 2% 3167Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 83m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 73m 0% 13723Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3278m 20% 14504Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14588Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 272m 1% 1952Mi 3% 21:58:21 DEBUG --- stderr --- 21:58:21 DEBUG 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG { 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG "data": { 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG "result": [ 21:59:01 DEBUG { 21:59:01 DEBUG "metric": { 21:59:01 DEBUG "pod": "ds-idrepo-2" 21:59:01 DEBUG }, 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG [...] 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG "pod": "ds-idrepo-0" 21:59:01 DEBUG }, 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG "value": [ 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG "636.8666666666667" 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG ] 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:01 DEBUG } 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG ] 21:59:01 DEBUG { 21:59:01 DEBUG } 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG } 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG "data": { 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG "resultType": "vector", 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG "result": [ 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:01 DEBUG { 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG "metric": { 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG { 21:59:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG }, 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG "data": { 21:59:01 DEBUG "value": [ 21:59:01 DEBUG { 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG "result": [ 21:59:01 DEBUG 21:59:01 DEBUG --- status code --- 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG "data": { 21:59:01 DEBUG { 21:59:01 DEBUG 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG "metric": { 21:59:01 DEBUG [...] 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG "result": [ 21:59:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:59:01 DEBUG 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG { 21:59:01 DEBUG }, 21:59:01 DEBUG 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:01 DEBUG "metric": { 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 21:59:01 DEBUG { 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:59:01 DEBUG 21:59:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG { 21:59:01 DEBUG }, 21:59:01 DEBUG 21:59:01 DEBUG }, 21:59:01 DEBUG "data": { 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 21:59:01 DEBUG "value": [ 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG "data": { 21:59:01 DEBUG 21:59:01 DEBUG [...] 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG "result": [ 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG "0" 21:59:01 DEBUG { 21:59:01 DEBUG "result": [ 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG ] 21:59:01 DEBUG "metric": { 21:59:01 DEBUG { 21:59:01 DEBUG [...] 21:59:01 DEBUG 21:59:01 DEBUG } 21:59:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:59:01 DEBUG "metric": { 21:59:01 DEBUG 21:59:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:59:01 DEBUG ] 21:59:01 DEBUG }, 21:59:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:59:01 DEBUG 21:59:01 DEBUG }, 21:59:01 DEBUG } 21:59:01 DEBUG "value": [ 21:59:01 DEBUG }, 21:59:01 DEBUG 21:59:01 DEBUG "value": [ 21:59:01 DEBUG } 21:59:01 DEBUG 21:59:01 DEBUG "value": [ 21:59:01 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG }, 21:59:01 DEBUG "0" 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG "value": [ 21:59:01 DEBUG ] 21:59:01 DEBUG [...] 21:59:01 DEBUG 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG } 21:59:01 DEBUG 21:59:01 DEBUG [...] 21:59:01 DEBUG "0.00001688041193721934" 21:59:01 DEBUG ] 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 21:59:01 DEBUG 21:59:01 DEBUG ] 21:59:01 DEBUG }, 21:59:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:59:01 DEBUG } 21:59:01 DEBUG "value": [ 21:59:01 DEBUG }, 21:59:01 DEBUG } 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG "value": [ 21:59:01 DEBUG "0" 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG ] 21:59:01 DEBUG "0.2" 21:59:01 DEBUG } 21:59:01 DEBUG ] 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG { 21:59:01 DEBUG { 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG "data": { 21:59:01 DEBUG "data": { 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG "result": [ 21:59:01 DEBUG "result": [ 21:59:01 DEBUG { 21:59:01 DEBUG { 21:59:01 DEBUG "metric": { 21:59:01 DEBUG "metric": { 21:59:01 DEBUG "pod": "ds-cts-0" 21:59:01 DEBUG "pod": "ds-cts-0" 21:59:01 DEBUG }, 21:59:01 DEBUG }, 21:59:01 DEBUG "value": [ 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG [...] 21:59:01 DEBUG [...] 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG "pod": "ds-idrepo-0" 21:59:01 DEBUG "pod": "ds-idrepo-0" 21:59:01 DEBUG }, 21:59:01 DEBUG }, 21:59:01 DEBUG "value": [ 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG "0" 21:59:01 DEBUG "636.7666666666667" 21:59:01 DEBUG ] 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 DEBUG ] 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG { 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG "data": { 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG "result": [ 21:59:01 DEBUG { 21:59:01 DEBUG "metric": { 21:59:01 DEBUG "pod": "ds-cts-0" 21:59:01 DEBUG }, 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG [...] 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG "pod": "ds-cts-2" 21:59:01 DEBUG }, 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG "0" 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG { 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG "data": { 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG "result": [ 21:59:01 DEBUG { 21:59:01 DEBUG "metric": { 21:59:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:59:01 DEBUG }, 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG [...] 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:59:01 DEBUG }, 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG "0" 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686689941" 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689941" 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG --- status code --- 21:59:01 INFO [http_cmd]: http status code OK 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG --- status code --- 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG http status code is 200 (expected 200) 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:01 DEBUG --- http response --- 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:01 DEBUG { 21:59:01 DEBUG ----- output ----- 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG { 21:59:01 DEBUG "data": { 21:59:01 DEBUG "status": "success", 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG "data": { 21:59:01 DEBUG "result": [ 21:59:01 DEBUG "resultType": "vector", 21:59:01 DEBUG { 21:59:01 DEBUG "result": [ 21:59:01 DEBUG "metric": { 21:59:01 DEBUG { 21:59:01 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:59:01 DEBUG "metric": { 21:59:01 DEBUG }, 21:59:01 DEBUG "instance": "10.142.0.100:9100" 21:59:01 DEBUG "value": [ 21:59:01 DEBUG }, 21:59:01 DEBUG 21:59:01 DEBUG "value": [ 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG [...] 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG [...] 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG 21:59:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 21:59:01 DEBUG 21:59:01 DEBUG }, 21:59:01 DEBUG "instance": "10.142.0.99:9100" 21:59:01 DEBUG "value": [ 21:59:01 DEBUG }, 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG "value": [ 21:59:01 DEBUG "13617192960" 21:59:01 DEBUG 1686689941, 21:59:01 DEBUG ] 21:59:01 DEBUG "0" 21:59:01 DEBUG } 21:59:01 DEBUG ] 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 DEBUG ] 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 DEBUG } 21:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689941" 21:59:02 INFO [http_cmd]: http status code OK 21:59:02 DEBUG --- status code --- 21:59:02 DEBUG http status code is 200 (expected 200) 21:59:02 DEBUG --- http response --- 21:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:59:02 DEBUG ----- output ----- 21:59:02 DEBUG { 21:59:02 DEBUG "status": "success", 21:59:02 DEBUG "data": { 21:59:02 DEBUG "resultType": "vector", 21:59:02 DEBUG "result": [ 21:59:02 DEBUG { 21:59:02 DEBUG "metric": { 21:59:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG [...] 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 1686689941, 21:59:02 DEBUG "0" 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG } 21:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689942" 21:59:02 INFO [http_cmd]: http status code OK 21:59:02 DEBUG --- status code --- 21:59:02 DEBUG http status code is 200 (expected 200) 21:59:02 DEBUG --- http response --- 21:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:02 DEBUG ----- output ----- 21:59:02 DEBUG { 21:59:02 DEBUG "status": "success", 21:59:02 DEBUG "data": { 21:59:02 DEBUG "resultType": "vector", 21:59:02 DEBUG "result": [ 21:59:02 DEBUG { 21:59:02 DEBUG "metric": { 21:59:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG [...] 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 1686689942, 21:59:02 DEBUG "0" 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG } 21:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689942" 21:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689942" 21:59:02 INFO [http_cmd]: http status code OK 21:59:02 DEBUG --- status code --- 21:59:02 DEBUG http status code is 200 (expected 200) 21:59:02 DEBUG --- http response --- 21:59:02 INFO [http_cmd]: http status code OK 21:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:02 DEBUG --- status code --- 21:59:02 DEBUG ----- output ----- 21:59:02 DEBUG http status code is 200 (expected 200) 21:59:02 DEBUG { 21:59:02 DEBUG --- http response --- 21:59:02 DEBUG "status": "success", 21:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:02 DEBUG "data": { 21:59:02 DEBUG ----- output ----- 21:59:02 DEBUG "resultType": "vector", 21:59:02 DEBUG { 21:59:02 DEBUG "result": [ 21:59:02 DEBUG "status": "success", 21:59:02 DEBUG { 21:59:02 DEBUG "data": { 21:59:02 DEBUG "metric": { 21:59:02 DEBUG "resultType": "vector", 21:59:02 DEBUG "pod": "ds-idrepo-2" 21:59:02 DEBUG "result": [ 21:59:02 DEBUG }, 21:59:02 DEBUG { 21:59:02 DEBUG "value": [ 21:59:02 DEBUG "metric": { 21:59:02 DEBUG 21:59:02 DEBUG "instance": "10.142.0.100:9100" 21:59:02 DEBUG 21:59:02 DEBUG }, 21:59:02 DEBUG 21:59:02 DEBUG "value": [ 21:59:02 DEBUG [...] 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG [...] 21:59:02 DEBUG "pod": "ds-idrepo-0" 21:59:02 DEBUG 21:59:02 DEBUG }, 21:59:02 DEBUG 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 21:59:02 DEBUG 1686689942, 21:59:02 DEBUG "instance": "10.142.0.99:9100" 21:59:02 DEBUG "636.9" 21:59:02 DEBUG }, 21:59:02 DEBUG ] 21:59:02 DEBUG "value": [ 21:59:02 DEBUG } 21:59:02 DEBUG 1686689942, 21:59:02 DEBUG ] 21:59:02 DEBUG "19895710.933333334" 21:59:02 DEBUG } 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG } 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG } 21:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689942" 21:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686689942" 21:59:02 INFO [http_cmd]: http status code OK 21:59:02 DEBUG --- status code --- 21:59:02 DEBUG http status code is 200 (expected 200) 21:59:02 DEBUG --- http response --- 21:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:02 DEBUG ----- output ----- 21:59:02 DEBUG { 21:59:02 DEBUG "status": "success", 21:59:02 DEBUG "data": { 21:59:02 DEBUG "resultType": "vector", 21:59:02 DEBUG "result": [ 21:59:02 DEBUG { 21:59:02 DEBUG "metric": { 21:59:02 DEBUG "instance": "10.142.0.100:9100" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG [...] 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG "instance": "10.142.0.99:9100" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 1686689942, 21:59:02 DEBUG "19893758.03333333" 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG } 21:59:02 INFO [http_cmd]: http status code OK 21:59:02 DEBUG --- status code --- 21:59:02 DEBUG http status code is 200 (expected 200) 21:59:02 DEBUG --- http response --- 21:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:02 DEBUG ----- output ----- 21:59:02 DEBUG { 21:59:02 DEBUG "status": "success", 21:59:02 DEBUG "data": { 21:59:02 DEBUG "resultType": "vector", 21:59:02 DEBUG "result": [ 21:59:02 DEBUG { 21:59:02 DEBUG "metric": { 21:59:02 DEBUG "instance": "10.142.0.100:9100" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG [...] 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG "instance": "10.142.0.99:9100" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 1686689942, 21:59:02 DEBUG "0.00036666666666747006" 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG } 21:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689942" 21:59:02 INFO [http_cmd]: http status code OK 21:59:02 DEBUG --- status code --- 21:59:02 DEBUG http status code is 200 (expected 200) 21:59:02 DEBUG --- http response --- 21:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:02 DEBUG ----- output ----- 21:59:02 DEBUG { 21:59:02 DEBUG "status": "success", 21:59:02 DEBUG "data": { 21:59:02 DEBUG "resultType": "vector", 21:59:02 DEBUG "result": [ 21:59:02 DEBUG { 21:59:02 DEBUG "metric": { 21:59:02 DEBUG "pod": "ds-idrepo-2" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG [...] 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG "pod": "ds-idrepo-0" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 1686689942, 21:59:02 DEBUG "636.9" 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG } 21:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686689942" 21:59:02 INFO [http_cmd]: http status code OK 21:59:02 DEBUG --- status code --- 21:59:02 DEBUG http status code is 200 (expected 200) 21:59:02 DEBUG --- http response --- 21:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:02 DEBUG ----- output ----- 21:59:02 DEBUG { 21:59:02 DEBUG "status": "success", 21:59:02 DEBUG "data": { 21:59:02 DEBUG "resultType": "vector", 21:59:02 DEBUG "result": [ 21:59:02 DEBUG { 21:59:02 DEBUG "metric": { 21:59:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG [...] 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 1686689942, 21:59:02 DEBUG "0.23359308097214598" 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG } 21:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686689942" 21:59:02 INFO [http_cmd]: http status code OK 21:59:02 DEBUG --- status code --- 21:59:02 DEBUG http status code is 200 (expected 200) 21:59:02 DEBUG --- http response --- 21:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:59:02 DEBUG ----- output ----- 21:59:02 DEBUG { 21:59:02 DEBUG "status": "success", 21:59:02 DEBUG "data": { 21:59:02 DEBUG "resultType": "vector", 21:59:02 DEBUG "result": [ 21:59:02 DEBUG { 21:59:02 DEBUG "metric": { 21:59:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG [...] 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG 21:59:02 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 21:59:02 DEBUG }, 21:59:02 DEBUG "value": [ 21:59:02 DEBUG 1686689942, 21:59:02 DEBUG "0" 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG ] 21:59:02 DEBUG } 21:59:02 DEBUG } 21:59:17 INFO 21:59:17 INFO [loop_until]: kubectl --namespace=xlou top pods 21:59:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:17 INFO [loop_until]: OK (rc = 0) 21:59:17 DEBUG --- stdout --- 21:59:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 8m 416Mi ds-cts-1 8m 424Mi ds-cts-2 8m 392Mi ds-idrepo-0 3425m 13829Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3423m 4871Mi idm-58dc667486-rgz9j 2465m 5093Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 212m 639Mi 21:59:17 DEBUG --- stderr --- 21:59:17 DEBUG 21:59:21 INFO 21:59:21 INFO [loop_until]: kubectl --namespace=xlou top node 21:59:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:21 INFO [loop_until]: OK (rc = 0) 21:59:21 DEBUG --- stdout --- 21:59:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3333m 20% 5820Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6813Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2612m 16% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 456m 2% 3156Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13711Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3447m 21% 14520Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 69m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 75m 0% 14589Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 278m 1% 1952Mi 3% 21:59:21 DEBUG --- stderr --- 21:59:21 DEBUG 22:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690001" 22:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690001" 22:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690001" 22:00:01 INFO [http_cmd]: http status code OK 22:00:01 DEBUG --- status code --- 22:00:01 DEBUG http status code is 200 (expected 200) 22:00:01 DEBUG --- http response --- 22:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:01 DEBUG ----- output ----- 22:00:01 DEBUG { 22:00:01 DEBUG "status": "success", 22:00:01 DEBUG "data": { 22:00:01 DEBUG "resultType": "vector", 22:00:01 DEBUG "result": [ 22:00:01 DEBUG { 22:00:01 DEBUG "metric": { 22:00:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:00:01 DEBUG }, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG [...] 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:00:01 DEBUG }, 22:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690001" 22:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690001" 22:00:01 DEBUG "value": [ 22:00:01 INFO [http_cmd]: http status code OK 22:00:01 DEBUG 1686690001, 22:00:01 DEBUG --- status code --- 22:00:01 INFO [http_cmd]: http status code OK 22:00:01 DEBUG "0" 22:00:01 DEBUG http status code is 200 (expected 200) 22:00:01 DEBUG --- status code --- 22:00:01 DEBUG ] 22:00:01 DEBUG --- http response --- 22:00:01 DEBUG http status code is 200 (expected 200) 22:00:01 DEBUG } 22:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:01 DEBUG --- http response --- 22:00:01 DEBUG ] 22:00:01 DEBUG ----- output ----- 22:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:00:01 DEBUG } 22:00:01 DEBUG { 22:00:01 DEBUG ----- output ----- 22:00:01 DEBUG } 22:00:01 DEBUG "status": "success", 22:00:01 DEBUG { 22:00:01 DEBUG "data": { 22:00:01 DEBUG "status": "success", 22:00:01 DEBUG "resultType": "vector", 22:00:01 DEBUG "data": { 22:00:01 DEBUG "result": [ 22:00:01 DEBUG "resultType": "vector", 22:00:01 DEBUG { 22:00:01 DEBUG "result": [ 22:00:01 DEBUG "metric": { 22:00:01 DEBUG { 22:00:01 DEBUG "pod": "ds-idrepo-2" 22:00:01 DEBUG "metric": { 22:00:01 DEBUG }, 22:00:01 DEBUG "pod": "ds-cts-0" 22:00:01 DEBUG "value": [ 22:00:01 DEBUG }, 22:00:01 DEBUG 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG [...] 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG [...] 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG "pod": "ds-idrepo-0" 22:00:01 DEBUG 22:00:01 DEBUG }, 22:00:01 DEBUG "pod": "ds-idrepo-0" 22:00:01 DEBUG "value": [ 22:00:01 DEBUG }, 22:00:01 DEBUG 1686690001, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG "671.2666666666667" 22:00:01 DEBUG 1686690001, 22:00:01 DEBUG ] 22:00:01 DEBUG "671.3333333333334" 22:00:01 DEBUG } 22:00:01 DEBUG ] 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 INFO [http_cmd]: http status code OK 22:00:01 DEBUG --- status code --- 22:00:01 DEBUG http status code is 200 (expected 200) 22:00:01 DEBUG --- http response --- 22:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:00:01 DEBUG ----- output ----- 22:00:01 DEBUG { 22:00:01 DEBUG "status": "success", 22:00:01 DEBUG "data": { 22:00:01 DEBUG "resultType": "vector", 22:00:01 DEBUG "result": [ 22:00:01 DEBUG { 22:00:01 DEBUG "metric": { 22:00:01 DEBUG "pod": "ds-cts-0" 22:00:01 DEBUG }, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG [...] 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG "pod": "ds-idrepo-0" 22:00:01 DEBUG }, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 1686690001, 22:00:01 DEBUG "0" 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 INFO [http_cmd]: http status code OK 22:00:01 DEBUG --- status code --- 22:00:01 DEBUG http status code is 200 (expected 200) 22:00:01 DEBUG --- http response --- 22:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:01 DEBUG ----- output ----- 22:00:01 DEBUG { 22:00:01 DEBUG "status": "success", 22:00:01 DEBUG "data": { 22:00:01 DEBUG "resultType": "vector", 22:00:01 DEBUG "result": [ 22:00:01 DEBUG { 22:00:01 DEBUG "metric": { 22:00:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:00:01 DEBUG }, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG [...] 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:00:01 DEBUG }, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 1686690001, 22:00:01 DEBUG "0.2" 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690001" 22:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690001" 22:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690001" 22:00:01 INFO [http_cmd]: http status code OK 22:00:01 INFO [http_cmd]: http status code OK 22:00:01 INFO [http_cmd]: http status code OK 22:00:01 DEBUG --- status code --- 22:00:01 DEBUG --- status code --- 22:00:01 DEBUG --- status code --- 22:00:01 DEBUG http status code is 200 (expected 200) 22:00:01 DEBUG http status code is 200 (expected 200) 22:00:01 DEBUG http status code is 200 (expected 200) 22:00:01 DEBUG --- http response --- 22:00:01 DEBUG --- http response --- 22:00:01 DEBUG --- http response --- 22:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:01 DEBUG ----- output ----- 22:00:01 DEBUG ----- output ----- 22:00:01 DEBUG ----- output ----- 22:00:01 DEBUG { 22:00:01 DEBUG { 22:00:01 DEBUG { 22:00:01 DEBUG "status": "success", 22:00:01 DEBUG "status": "success", 22:00:01 DEBUG "status": "success", 22:00:01 DEBUG "data": { 22:00:01 DEBUG "data": { 22:00:01 DEBUG "data": { 22:00:01 DEBUG "resultType": "vector", 22:00:01 DEBUG "resultType": "vector", 22:00:01 DEBUG "resultType": "vector", 22:00:01 DEBUG "result": [ 22:00:01 DEBUG "result": [ 22:00:01 DEBUG "result": [ 22:00:01 DEBUG { 22:00:01 DEBUG { 22:00:01 DEBUG { 22:00:01 DEBUG "metric": { 22:00:01 DEBUG "metric": { 22:00:01 DEBUG "metric": { 22:00:01 DEBUG "pod": "ds-cts-2" 22:00:01 DEBUG "pod": "ds-cts-2" 22:00:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:00:01 DEBUG }, 22:00:01 DEBUG }, 22:00:01 DEBUG }, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG "value": [ 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG [...] 22:00:01 DEBUG [...] 22:00:01 DEBUG [...] 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:00:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:00:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:00:01 DEBUG }, 22:00:01 DEBUG }, 22:00:01 DEBUG }, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG "value": [ 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 1686690001, 22:00:01 DEBUG 1686690001, 22:00:01 DEBUG 1686690001, 22:00:01 DEBUG "0.00003478353026721988" 22:00:01 DEBUG "0" 22:00:01 DEBUG "0" 22:00:01 DEBUG ] 22:00:01 DEBUG ] 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 DEBUG ] 22:00:01 DEBUG ] 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690001" 22:00:01 INFO [http_cmd]: http status code OK 22:00:01 DEBUG --- status code --- 22:00:01 DEBUG http status code is 200 (expected 200) 22:00:01 DEBUG --- http response --- 22:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:00:01 DEBUG ----- output ----- 22:00:01 DEBUG { 22:00:01 DEBUG "status": "success", 22:00:01 DEBUG "data": { 22:00:01 DEBUG "resultType": "vector", 22:00:01 DEBUG "result": [ 22:00:01 DEBUG { 22:00:01 DEBUG "metric": { 22:00:01 DEBUG "pod": "ds-cts-0" 22:00:01 DEBUG }, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG [...] 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG 22:00:01 DEBUG "pod": "ds-cts-2" 22:00:01 DEBUG }, 22:00:01 DEBUG "value": [ 22:00:01 DEBUG 1686690001, 22:00:01 DEBUG "0" 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG ] 22:00:01 DEBUG } 22:00:01 DEBUG } 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690002" 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690002" 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG "0" 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG ] 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG } 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:02 DEBUG ] 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG } 22:00:02 DEBUG { 22:00:02 DEBUG } 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "instance": "10.142.0.100:9100" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "instance": "10.142.0.99:9100" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "0" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690002" 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "13617061888" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690002" 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "pod": "ds-cts-2" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "0" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690002" 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "0" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690002" 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690002" 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "pod": "ds-idrepo-2" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "pod": "ds-idrepo-0" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "671.3333333333334" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "instance": "10.142.0.100:9100" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "instance": "10.142.0.99:9100" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "19306898.266666666" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690002" 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690002" 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "instance": "10.142.0.100:9100" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "instance": "10.142.0.99:9100" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "19304521.96666667" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "instance": "10.142.0.100:9100" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "instance": "10.142.0.99:9100" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "0.00040000000000001514" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690002" 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "pod": "ds-idrepo-2" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "pod": "ds-idrepo-0" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "671.2333333333333" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690002" 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "0.22368724451318114" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690002" 22:00:02 INFO [http_cmd]: http status code OK 22:00:02 DEBUG --- status code --- 22:00:02 DEBUG http status code is 200 (expected 200) 22:00:02 DEBUG --- http response --- 22:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:00:02 DEBUG ----- output ----- 22:00:02 DEBUG { 22:00:02 DEBUG "status": "success", 22:00:02 DEBUG "data": { 22:00:02 DEBUG "resultType": "vector", 22:00:02 DEBUG "result": [ 22:00:02 DEBUG { 22:00:02 DEBUG "metric": { 22:00:02 DEBUG "pod": "ds-cts-2" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG [...] 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG 22:00:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:00:02 DEBUG }, 22:00:02 DEBUG "value": [ 22:00:02 DEBUG 1686690002, 22:00:02 DEBUG "0" 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG ] 22:00:02 DEBUG } 22:00:02 DEBUG } 22:00:17 INFO 22:00:17 INFO [loop_until]: kubectl --namespace=xlou top pods 22:00:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:00:17 INFO [loop_until]: OK (rc = 0) 22:00:17 DEBUG --- stdout --- 22:00:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 392Mi ds-idrepo-0 3322m 13806Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3173m 4875Mi idm-58dc667486-rgz9j 2614m 5093Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 204m 639Mi 22:00:18 DEBUG --- stderr --- 22:00:18 DEBUG 22:00:21 INFO 22:00:21 INFO [loop_until]: kubectl --namespace=xlou top node 22:00:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:00:21 INFO [loop_until]: OK (rc = 0) 22:00:21 DEBUG --- stdout --- 22:00:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3392m 21% 5824Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6814Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2775m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 523m 3% 3155Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 83m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1195Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13712Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3497m 22% 14517Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14592Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 275m 1% 1955Mi 3% 22:00:21 DEBUG --- stderr --- 22:00:21 DEBUG 22:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690061" 22:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690061" 22:01:01 INFO [http_cmd]: http status code OK 22:01:01 DEBUG --- status code --- 22:01:01 DEBUG http status code is 200 (expected 200) 22:01:01 DEBUG --- http response --- 22:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:01 DEBUG ----- output ----- 22:01:01 DEBUG { 22:01:01 DEBUG "status": "success", 22:01:01 DEBUG "data": { 22:01:01 DEBUG "resultType": "vector", 22:01:01 DEBUG "result": [ 22:01:01 DEBUG { 22:01:01 DEBUG "metric": { 22:01:01 DEBUG "pod": "ds-idrepo-2" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG [...] 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG "pod": "ds-idrepo-0" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 1686690061, 22:01:01 DEBUG "598.7333333333333" 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG } 22:01:01 INFO [http_cmd]: http status code OK 22:01:01 DEBUG --- status code --- 22:01:01 DEBUG http status code is 200 (expected 200) 22:01:01 DEBUG --- http response --- 22:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:01 DEBUG ----- output ----- 22:01:01 DEBUG { 22:01:01 DEBUG "status": "success", 22:01:01 DEBUG "data": { 22:01:01 DEBUG "resultType": "vector", 22:01:01 DEBUG "result": [ 22:01:01 DEBUG { 22:01:01 DEBUG "metric": { 22:01:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG [...] 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 1686690061, 22:01:01 DEBUG "0.2" 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG } 22:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690061" 22:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690061" 22:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690061" 22:01:01 INFO [http_cmd]: http status code OK 22:01:01 DEBUG --- status code --- 22:01:01 DEBUG http status code is 200 (expected 200) 22:01:01 DEBUG --- http response --- 22:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:01 DEBUG ----- output ----- 22:01:01 DEBUG { 22:01:01 DEBUG "status": "success", 22:01:01 DEBUG "data": { 22:01:01 DEBUG "resultType": "vector", 22:01:01 DEBUG "result": [ 22:01:01 DEBUG { 22:01:01 DEBUG "metric": { 22:01:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG [...] 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 1686690061, 22:01:01 DEBUG "0" 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG } 22:01:01 INFO [http_cmd]: http status code OK 22:01:01 DEBUG --- status code --- 22:01:01 DEBUG http status code is 200 (expected 200) 22:01:01 DEBUG --- http response --- 22:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:01 DEBUG ----- output ----- 22:01:01 DEBUG { 22:01:01 DEBUG "status": "success", 22:01:01 DEBUG "data": { 22:01:01 DEBUG "resultType": "vector", 22:01:01 DEBUG "result": [ 22:01:01 DEBUG { 22:01:01 DEBUG "metric": { 22:01:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG [...] 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 1686690061, 22:01:01 DEBUG "0" 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG } 22:01:01 INFO [http_cmd]: http status code OK 22:01:01 DEBUG --- status code --- 22:01:01 DEBUG http status code is 200 (expected 200) 22:01:01 DEBUG --- http response --- 22:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:01:01 DEBUG ----- output ----- 22:01:01 DEBUG { 22:01:01 DEBUG "status": "success", 22:01:01 DEBUG "data": { 22:01:01 DEBUG "resultType": "vector", 22:01:01 DEBUG "result": [ 22:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690061" 22:01:01 DEBUG { 22:01:01 DEBUG "metric": { 22:01:01 DEBUG "pod": "ds-cts-0" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG [...] 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG "pod": "ds-idrepo-0" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 1686690061, 22:01:01 DEBUG "598.7333333333333" 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG } 22:01:01 INFO [http_cmd]: http status code OK 22:01:01 DEBUG --- status code --- 22:01:01 DEBUG http status code is 200 (expected 200) 22:01:01 DEBUG --- http response --- 22:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:01:01 DEBUG ----- output ----- 22:01:01 DEBUG { 22:01:01 DEBUG "status": "success", 22:01:01 DEBUG "data": { 22:01:01 DEBUG "resultType": "vector", 22:01:01 DEBUG "result": [ 22:01:01 DEBUG { 22:01:01 DEBUG "metric": { 22:01:01 DEBUG "pod": "ds-cts-0" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG [...] 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG "pod": "ds-idrepo-0" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 1686690061, 22:01:01 DEBUG "0" 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG } 22:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690061" 22:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690061" 22:01:01 INFO [http_cmd]: http status code OK 22:01:01 DEBUG --- status code --- 22:01:01 DEBUG http status code is 200 (expected 200) 22:01:01 DEBUG --- http response --- 22:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:01 DEBUG ----- output ----- 22:01:01 DEBUG { 22:01:01 DEBUG "status": "success", 22:01:01 DEBUG "data": { 22:01:01 DEBUG "resultType": "vector", 22:01:01 DEBUG "result": [ 22:01:01 DEBUG { 22:01:01 DEBUG "metric": { 22:01:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG [...] 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 1686690061, 22:01:01 DEBUG "0" 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG } 22:01:01 INFO [http_cmd]: http status code OK 22:01:01 DEBUG --- status code --- 22:01:01 DEBUG http status code is 200 (expected 200) 22:01:01 DEBUG --- http response --- 22:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:01 DEBUG ----- output ----- 22:01:01 DEBUG { 22:01:01 DEBUG "status": "success", 22:01:01 DEBUG "data": { 22:01:01 DEBUG "resultType": "vector", 22:01:01 DEBUG "result": [ 22:01:01 DEBUG { 22:01:01 DEBUG "metric": { 22:01:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG [...] 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG 22:01:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:01:01 DEBUG }, 22:01:01 DEBUG "value": [ 22:01:01 DEBUG 1686690061, 22:01:01 DEBUG "0" 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG ] 22:01:01 DEBUG } 22:01:01 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "pod": "ds-cts-0" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "pod": "ds-cts-2" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "0" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690062" 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG "0" 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG ] 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG } 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:02 DEBUG ] 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG } 22:01:02 DEBUG { 22:01:02 DEBUG } 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "instance": "10.142.0.100:9100" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "instance": "10.142.0.99:9100" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "0" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "13619585024" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "0" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "0" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690062" 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "pod": "ds-idrepo-2" 22:01:02 DEBUG "instance": "10.142.0.100:9100" 22:01:02 DEBUG }, 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "pod": "ds-idrepo-0" 22:01:02 DEBUG "instance": "10.142.0.99:9100" 22:01:02 DEBUG }, 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "598.7333333333333" 22:01:02 DEBUG "18801182.066666666" 22:01:02 DEBUG ] 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690062" 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "instance": "10.142.0.100:9100" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "instance": "10.142.0.99:9100" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "0.00046666666666510536" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "instance": "10.142.0.100:9100" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "instance": "10.142.0.99:9100" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "18800923.599999998" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "pod": "ds-idrepo-2" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "pod": "ds-idrepo-0" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "598.7333333333333" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "0.22525610627426187" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690062" 22:01:02 INFO [http_cmd]: http status code OK 22:01:02 DEBUG --- status code --- 22:01:02 DEBUG http status code is 200 (expected 200) 22:01:02 DEBUG --- http response --- 22:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:01:02 DEBUG ----- output ----- 22:01:02 DEBUG { 22:01:02 DEBUG "status": "success", 22:01:02 DEBUG "data": { 22:01:02 DEBUG "resultType": "vector", 22:01:02 DEBUG "result": [ 22:01:02 DEBUG { 22:01:02 DEBUG "metric": { 22:01:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG [...] 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG 22:01:02 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 22:01:02 DEBUG }, 22:01:02 DEBUG "value": [ 22:01:02 DEBUG 1686690062, 22:01:02 DEBUG "0" 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG ] 22:01:02 DEBUG } 22:01:02 DEBUG } 22:01:18 INFO 22:01:18 INFO [loop_until]: kubectl --namespace=xlou top pods 22:01:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:01:18 INFO [loop_until]: OK (rc = 0) 22:01:18 DEBUG --- stdout --- 22:01:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5698Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 424Mi ds-cts-2 9m 393Mi ds-idrepo-0 3363m 13822Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 17m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3292m 4878Mi idm-58dc667486-rgz9j 2351m 5093Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 202m 639Mi 22:01:18 DEBUG --- stderr --- 22:01:18 DEBUG 22:01:21 INFO 22:01:21 INFO [loop_until]: kubectl --namespace=xlou top node 22:01:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:01:21 INFO [loop_until]: OK (rc = 0) 22:01:21 DEBUG --- stdout --- 22:01:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3393m 21% 5823Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 72m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 71m 0% 6816Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2475m 15% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 460m 2% 3144Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13714Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3226m 20% 14503Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14593Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 269m 1% 1956Mi 3% 22:01:21 DEBUG --- stderr --- 22:01:21 DEBUG 22:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690121" 22:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690121" 22:02:01 INFO [http_cmd]: http status code OK 22:02:01 INFO [http_cmd]: http status code OK 22:02:01 DEBUG --- status code --- 22:02:01 DEBUG --- status code --- 22:02:01 DEBUG http status code is 200 (expected 200) 22:02:01 DEBUG http status code is 200 (expected 200) 22:02:01 DEBUG --- http response --- 22:02:01 DEBUG --- http response --- 22:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:01 DEBUG ----- output ----- 22:02:01 DEBUG ----- output ----- 22:02:01 DEBUG { 22:02:01 DEBUG { 22:02:01 DEBUG "status": "success", 22:02:01 DEBUG "status": "success", 22:02:01 DEBUG "data": { 22:02:01 DEBUG "data": { 22:02:01 DEBUG "resultType": "vector", 22:02:01 DEBUG "resultType": "vector", 22:02:01 DEBUG "result": [ 22:02:01 DEBUG "result": [ 22:02:01 DEBUG { 22:02:01 DEBUG { 22:02:01 DEBUG "metric": { 22:02:01 DEBUG "metric": { 22:02:01 DEBUG "pod": "ds-idrepo-2" 22:02:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:02:01 DEBUG }, 22:02:01 DEBUG }, 22:02:01 DEBUG "value": [ 22:02:01 DEBUG "value": [ 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG [...] 22:02:01 DEBUG [...] 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG "pod": "ds-idrepo-0" 22:02:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:02:01 DEBUG }, 22:02:01 DEBUG }, 22:02:01 DEBUG "value": [ 22:02:01 DEBUG "value": [ 22:02:01 DEBUG 1686690121, 22:02:01 DEBUG 1686690121, 22:02:01 DEBUG "442.1333333333333" 22:02:01 DEBUG "0" 22:02:01 DEBUG ] 22:02:01 DEBUG ] 22:02:01 DEBUG } 22:02:01 DEBUG } 22:02:01 DEBUG ] 22:02:01 DEBUG ] 22:02:01 DEBUG } 22:02:01 DEBUG } 22:02:01 DEBUG } 22:02:01 DEBUG } 22:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690121" 22:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690121" 22:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690121" 22:02:01 INFO [http_cmd]: http status code OK 22:02:01 INFO [http_cmd]: http status code OK 22:02:01 DEBUG --- status code --- 22:02:01 DEBUG --- status code --- 22:02:01 DEBUG http status code is 200 (expected 200) 22:02:01 DEBUG http status code is 200 (expected 200) 22:02:01 DEBUG --- http response --- 22:02:01 DEBUG --- http response --- 22:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:02:01 DEBUG ----- output ----- 22:02:01 DEBUG ----- output ----- 22:02:01 DEBUG { 22:02:01 DEBUG { 22:02:01 DEBUG "status": "success", 22:02:01 DEBUG "status": "success", 22:02:01 DEBUG "data": { 22:02:01 DEBUG "data": { 22:02:01 DEBUG "resultType": "vector", 22:02:01 DEBUG "resultType": "vector", 22:02:01 DEBUG "result": [ 22:02:01 DEBUG "result": [ 22:02:01 DEBUG { 22:02:01 DEBUG { 22:02:01 DEBUG "metric": { 22:02:01 DEBUG "metric": { 22:02:01 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:02:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:02:01 DEBUG }, 22:02:01 DEBUG }, 22:02:01 DEBUG "value": [ 22:02:01 DEBUG "value": [ 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG [...] 22:02:01 DEBUG [...] 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:02:01 DEBUG "pod": "idm-58dc667486-rgz9j" 22:02:01 DEBUG }, 22:02:01 DEBUG }, 22:02:01 DEBUG "value": [ 22:02:01 DEBUG "value": [ 22:02:01 DEBUG 1686690121, 22:02:01 DEBUG 1686690121, 22:02:01 DEBUG "0.2" 22:02:01 DEBUG "0" 22:02:01 DEBUG ] 22:02:01 DEBUG ] 22:02:01 DEBUG } 22:02:01 DEBUG } 22:02:01 DEBUG ] 22:02:01 DEBUG ] 22:02:01 DEBUG } 22:02:01 DEBUG } 22:02:01 DEBUG } 22:02:01 DEBUG } 22:02:01 INFO [http_cmd]: http status code OK 22:02:01 DEBUG --- status code --- 22:02:01 DEBUG http status code is 200 (expected 200) 22:02:01 DEBUG --- http response --- 22:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:02:01 DEBUG ----- output ----- 22:02:01 DEBUG { 22:02:01 DEBUG "status": "success", 22:02:01 DEBUG "data": { 22:02:01 DEBUG "resultType": "vector", 22:02:01 DEBUG "result": [ 22:02:01 DEBUG { 22:02:01 DEBUG "metric": { 22:02:01 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:02:01 DEBUG }, 22:02:01 DEBUG "value": [ 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG [...] 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG 22:02:01 DEBUG "pod": "idm-58dc667486-rgz9j" 22:02:01 DEBUG }, 22:02:01 DEBUG "value": [ 22:02:01 DEBUG 1686690121, 22:02:01 DEBUG "0" 22:02:01 DEBUG ] 22:02:01 DEBUG } 22:02:01 DEBUG ] 22:02:01 DEBUG } 22:02:01 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "pod": "ds-cts-0" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "pod": "ds-idrepo-0" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "442.09999999999997" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "pod": "ds-cts-0" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "pod": "ds-idrepo-0" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "0" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "0" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "pod": "ds-cts-0" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "pod": "ds-cts-2" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "0" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690122" 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "instance": "10.142.0.100:9100" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "instance": "10.142.0.99:9100" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "0" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "13616480256" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "0" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "pod": "ds-idrepo-0" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "2436.7531460554937" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "0" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "pod": "ds-idrepo-2" 22:02:02 DEBUG "instance": "10.142.0.100:9100" 22:02:02 DEBUG }, 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "pod": "ds-idrepo-0" 22:02:02 DEBUG "instance": "10.142.0.99:9100" 22:02:02 DEBUG }, 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "442.1666666666667" 22:02:02 DEBUG "12884935.266666666" 22:02:02 DEBUG ] 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690122" 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "instance": "10.142.0.100:9100" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "instance": "10.142.0.99:9100" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "0.00040000000000001514" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "instance": "10.142.0.100:9100" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "instance": "10.142.0.99:9100" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "12883107.666666666" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "pod": "ds-idrepo-2" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "pod": "ds-idrepo-0" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "442.1666666666667" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690122" 22:02:02 INFO [http_cmd]: http status code OK 22:02:02 DEBUG --- status code --- 22:02:02 DEBUG http status code is 200 (expected 200) 22:02:02 DEBUG --- http response --- 22:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:02 DEBUG ----- output ----- 22:02:02 DEBUG { 22:02:02 DEBUG "status": "success", 22:02:02 DEBUG "data": { 22:02:02 DEBUG "resultType": "vector", 22:02:02 DEBUG "result": [ 22:02:02 DEBUG { 22:02:02 DEBUG "metric": { 22:02:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG [...] 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG 22:02:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:02:02 DEBUG }, 22:02:02 DEBUG "value": [ 22:02:02 DEBUG 1686690122, 22:02:02 DEBUG "0.18797378821606509" 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG ] 22:02:02 DEBUG } 22:02:02 DEBUG } 22:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690122" 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG http status code is 200 (expected 200) 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG { 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "data": { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "result": [ 22:02:03 DEBUG { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:02:03 DEBUG }, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "pod": "ds-idrepo-0" 22:02:03 DEBUG }, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 1686690122, 22:02:03 DEBUG "0.3865510106636957" 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:18 INFO 22:02:18 INFO [loop_until]: kubectl --namespace=xlou top pods 22:02:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:02:18 INFO [loop_until]: OK (rc = 0) 22:02:18 DEBUG --- stdout --- 22:02:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 6m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 392Mi ds-idrepo-0 3573m 13810Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3270m 4880Mi idm-58dc667486-rgz9j 2580m 5092Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 228m 639Mi 22:02:18 DEBUG --- stderr --- 22:02:18 DEBUG 22:02:21 INFO 22:02:21 INFO [loop_until]: kubectl --namespace=xlou top node 22:02:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:02:21 INFO [loop_until]: OK (rc = 0) 22:02:21 DEBUG --- stdout --- 22:02:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3583m 22% 5829Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6825Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2743m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 69m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 454m 2% 3132Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13712Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3533m 22% 14508Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 60m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14593Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 302m 1% 1956Mi 3% 22:02:21 DEBUG --- stderr --- 22:02:21 DEBUG 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:03:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:03:02 DEBUG }, 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:03:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:03:02 DEBUG }, 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "0" 22:03:02 DEBUG "0" 22:03:02 DEBUG ] 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "ds-idrepo-2" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "ds-idrepo-0" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "580.7333333333333" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:03:02 DEBUG { 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG { 22:03:02 DEBUG "data": { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "result": [ 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG { 22:03:02 DEBUG "result": [ 22:03:02 DEBUG "metric": { 22:03:02 DEBUG { 22:03:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:03:02 DEBUG "metric": { 22:03:02 DEBUG }, 22:03:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:03:02 DEBUG "value": [ 22:03:02 DEBUG }, 22:03:02 DEBUG 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG 22:03:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG 22:03:02 DEBUG }, 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:03:02 DEBUG "value": [ 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:02 DEBUG }, 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG "value": [ 22:03:02 DEBUG "0.2" 22:03:02 DEBUG { 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG ] 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "0" 22:03:02 DEBUG } 22:03:02 DEBUG "data": { 22:03:02 DEBUG ] 22:03:02 DEBUG ] 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 DEBUG "result": [ 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG { 22:03:02 DEBUG } 22:03:02 DEBUG "metric": { 22:03:02 DEBUG } 22:03:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "0" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "ds-cts-0" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "ds-cts-2" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "0" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "ds-cts-0" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "ds-idrepo-0" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "580.8" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "ds-cts-0" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "ds-idrepo-0" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "0" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690182" 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "instance": "10.142.0.100:9100" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "instance": "10.142.0.99:9100" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "0" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "13616693248" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "0" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "ds-idrepo-0" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "4464.6634467951535" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "0" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "ds-idrepo-2" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "ds-idrepo-0" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "580.7333333333333" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "instance": "10.142.0.100:9100" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "instance": "10.142.0.99:9100" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "19976677.7" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690182" 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "instance": "10.142.0.100:9100" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "instance": "10.142.0.99:9100" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "0.00036666666666747006" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "instance": "10.142.0.100:9100" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "instance": "10.142.0.99:9100" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "19976638.833333332" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "pod": "ds-idrepo-2" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "pod": "ds-idrepo-0" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "580.7333333333333" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690182" 22:03:02 INFO [http_cmd]: http status code OK 22:03:02 DEBUG --- status code --- 22:03:02 DEBUG http status code is 200 (expected 200) 22:03:02 DEBUG --- http response --- 22:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:02 DEBUG ----- output ----- 22:03:02 DEBUG { 22:03:02 DEBUG "status": "success", 22:03:02 DEBUG "data": { 22:03:02 DEBUG "resultType": "vector", 22:03:02 DEBUG "result": [ 22:03:02 DEBUG { 22:03:02 DEBUG "metric": { 22:03:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG [...] 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG 22:03:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:03:02 DEBUG }, 22:03:02 DEBUG "value": [ 22:03:02 DEBUG 1686690182, 22:03:02 DEBUG "0.24719823693235843" 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG ] 22:03:02 DEBUG } 22:03:02 DEBUG } 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690183" 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG http status code is 200 (expected 200) 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG { 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "data": { 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "result": [ 22:03:03 DEBUG { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:03:03 DEBUG }, 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:03:03 DEBUG }, 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 1686690183, 22:03:03 DEBUG "0" 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:18 INFO 22:03:18 INFO [loop_until]: kubectl --namespace=xlou top pods 22:03:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:03:18 INFO [loop_until]: OK (rc = 0) 22:03:18 DEBUG --- stdout --- 22:03:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 424Mi ds-cts-2 8m 392Mi ds-idrepo-0 3065m 13805Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2812m 4881Mi idm-58dc667486-rgz9j 2460m 5092Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 210m 640Mi 22:03:18 DEBUG --- stderr --- 22:03:18 DEBUG 22:03:21 INFO 22:03:21 INFO [loop_until]: kubectl --namespace=xlou top node 22:03:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:03:21 INFO [loop_until]: OK (rc = 0) 22:03:21 DEBUG --- stdout --- 22:03:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3046m 19% 5829Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6825Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2512m 15% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 432m 2% 3137Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 83m 0% 1482Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 58m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1195Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13715Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3287m 20% 14516Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14592Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 278m 1% 1955Mi 3% 22:03:21 DEBUG --- stderr --- 22:03:21 DEBUG 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "173532004.0722534" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:04:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:04:02 DEBUG }, 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:04:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:04:02 DEBUG }, 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0" 22:04:02 DEBUG "0.2" 22:04:02 DEBUG ] 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0.006213432211524861" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG } 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "ds-idrepo-2" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "ds-idrepo-0" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "600.3333333333334" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "ds-cts-0" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "ds-idrepo-0" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "ds-cts-0" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "ds-cts-2" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "ds-cts-0" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "ds-idrepo-0" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "600.3333333333334" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690242" 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "instance": "10.142.0.100:9100" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "instance": "10.142.0.99:9100" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "13616734208" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "ds-idrepo-2" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "ds-idrepo-0" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "600.3666666666667" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "instance": "10.142.0.100:9100" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "instance": "10.142.0.99:9100" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "19152045.53333333" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690242" 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "instance": "10.142.0.100:9100" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "instance": "10.142.0.99:9100" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "19149861.03333333" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "instance": "10.142.0.100:9100" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "instance": "10.142.0.99:9100" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "0.00036666666666557526" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690242" 22:04:02 INFO [http_cmd]: http status code OK 22:04:02 DEBUG --- status code --- 22:04:02 DEBUG http status code is 200 (expected 200) 22:04:02 DEBUG --- http response --- 22:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:02 DEBUG ----- output ----- 22:04:02 DEBUG { 22:04:02 DEBUG "status": "success", 22:04:02 DEBUG "data": { 22:04:02 DEBUG "resultType": "vector", 22:04:02 DEBUG "result": [ 22:04:02 DEBUG { 22:04:02 DEBUG "metric": { 22:04:02 DEBUG "pod": "ds-idrepo-2" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG [...] 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG 22:04:02 DEBUG "pod": "ds-idrepo-0" 22:04:02 DEBUG }, 22:04:02 DEBUG "value": [ 22:04:02 DEBUG 1686690242, 22:04:02 DEBUG "600.4" 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG ] 22:04:02 DEBUG } 22:04:02 DEBUG } 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690243" 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG http status code is 200 (expected 200) 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG { 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "data": { 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "result": [ 22:04:03 DEBUG { 22:04:03 DEBUG "metric": { 22:04:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 1686690243, 22:04:03 DEBUG "0.236592889161804" 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690243" 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG http status code is 200 (expected 200) 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG { 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "data": { 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "result": [ 22:04:03 DEBUG { 22:04:03 DEBUG "metric": { 22:04:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "pod": "lodemon-6455d5ff7c-zzndq" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 1686690243, 22:04:03 DEBUG "0" 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:18 INFO 22:04:18 INFO [loop_until]: kubectl --namespace=xlou top pods 22:04:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:04:18 INFO [loop_until]: OK (rc = 0) 22:04:18 DEBUG --- stdout --- 22:04:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 10m 425Mi ds-cts-2 9m 392Mi ds-idrepo-0 3407m 13805Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3277m 4883Mi idm-58dc667486-rgz9j 2821m 5092Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 228m 639Mi 22:04:18 DEBUG --- stderr --- 22:04:18 DEBUG 22:04:21 INFO 22:04:21 INFO [loop_until]: kubectl --namespace=xlou top node 22:04:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:04:22 INFO [loop_until]: OK (rc = 0) 22:04:22 DEBUG --- stdout --- 22:04:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3439m 21% 5832Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 77m 0% 6826Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 3114m 19% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 472m 2% 3139Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 66m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13715Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3640m 22% 14497Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14593Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 290m 1% 1957Mi 3% 22:04:22 DEBUG --- stderr --- 22:04:22 DEBUG 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "idm-58dc667486-rgz9j" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "43935.73486668565" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0.2" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "ds-idrepo-2" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "ds-idrepo-0" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "603.2666666666667" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "idm-58dc667486-rgz9j" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "1.3112220625938609" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "ds-cts-0" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "ds-idrepo-0" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "603.2666666666667" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "ds-cts-0" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "ds-cts-2" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "ds-cts-0" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "ds-idrepo-0" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "instance": "10.142.0.100:9100" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "instance": "10.142.0.99:9100" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "13618778112" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "idm-58dc667486-rgz9j" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "ds-idrepo-2" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "ds-idrepo-0" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "603.2666666666667" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "instance": "10.142.0.100:9100" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "instance": "10.142.0.99:9100" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "18395758.633333333" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "instance": "10.142.0.100:9100" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "instance": "10.142.0.99:9100" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "18393930" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "instance": "10.142.0.100:9100" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "instance": "10.142.0.99:9100" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "0.0003333333333330302" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690302" 22:05:02 INFO [http_cmd]: http status code OK 22:05:02 DEBUG --- status code --- 22:05:02 DEBUG http status code is 200 (expected 200) 22:05:02 DEBUG --- http response --- 22:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:02 DEBUG ----- output ----- 22:05:02 DEBUG { 22:05:02 DEBUG "status": "success", 22:05:02 DEBUG "data": { 22:05:02 DEBUG "resultType": "vector", 22:05:02 DEBUG "result": [ 22:05:02 DEBUG { 22:05:02 DEBUG "metric": { 22:05:02 DEBUG "pod": "ds-idrepo-2" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG [...] 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG 22:05:02 DEBUG "pod": "ds-idrepo-0" 22:05:02 DEBUG }, 22:05:02 DEBUG "value": [ 22:05:02 DEBUG 1686690302, 22:05:02 DEBUG "603.2666666666667" 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG ] 22:05:02 DEBUG } 22:05:02 DEBUG } 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690303" 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG http status code is 200 (expected 200) 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "data": { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1686690303, 22:05:03 DEBUG "0.2441189728122121" 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690303" 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG http status code is 200 (expected 200) 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "data": { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1686690303, 22:05:03 DEBUG "0" 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:18 INFO 22:05:18 INFO [loop_until]: kubectl --namespace=xlou top pods 22:05:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:05:18 INFO [loop_until]: OK (rc = 0) 22:05:18 DEBUG --- stdout --- 22:05:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 424Mi ds-cts-2 8m 393Mi ds-idrepo-0 3331m 13799Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3269m 4886Mi idm-58dc667486-rgz9j 2510m 5092Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 223m 639Mi 22:05:18 DEBUG --- stderr --- 22:05:18 DEBUG 22:05:22 INFO 22:05:22 INFO [loop_until]: kubectl --namespace=xlou top node 22:05:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:05:22 INFO [loop_until]: OK (rc = 0) 22:05:22 DEBUG --- stdout --- 22:05:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3192m 20% 5834Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6852Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2472m 15% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6946Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 436m 2% 3135Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 67m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13711Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3487m 21% 14502Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14595Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 297m 1% 1955Mi 3% 22:05:22 DEBUG --- stderr --- 22:05:22 DEBUG 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "ds-idrepo-2" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "ds-idrepo-0" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "635.2666666666667" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0.00003081361383701421" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0.2" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "ds-cts-0" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "ds-idrepo-0" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "635.2666666666667" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "ds-cts-0" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "ds-cts-2" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "ds-cts-0" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "ds-idrepo-0" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "instance": "10.142.0.100:9100" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "instance": "10.142.0.99:9100" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0.0003333333333333262" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "13617057792" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "ds-idrepo-2" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "ds-idrepo-0" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "635.2666666666667" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "instance": "10.142.0.100:9100" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "instance": "10.142.0.99:9100" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "22497617" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "instance": "10.142.0.100:9100" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "instance": "10.142.0.99:9100" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "22495783.766666666" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "instance": "10.142.0.100:9100" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "instance": "10.142.0.99:9100" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "0.00026666666666793997" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690362" 22:06:02 INFO [http_cmd]: http status code OK 22:06:02 DEBUG --- status code --- 22:06:02 DEBUG http status code is 200 (expected 200) 22:06:02 DEBUG --- http response --- 22:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:02 DEBUG ----- output ----- 22:06:02 DEBUG { 22:06:02 DEBUG "status": "success", 22:06:02 DEBUG "data": { 22:06:02 DEBUG "resultType": "vector", 22:06:02 DEBUG "result": [ 22:06:02 DEBUG { 22:06:02 DEBUG "metric": { 22:06:02 DEBUG "pod": "ds-idrepo-2" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG [...] 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG 22:06:02 DEBUG "pod": "ds-idrepo-0" 22:06:02 DEBUG }, 22:06:02 DEBUG "value": [ 22:06:02 DEBUG 1686690362, 22:06:02 DEBUG "635.2666666666667" 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG ] 22:06:02 DEBUG } 22:06:02 DEBUG } 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690363" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected 200) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1686690363, 22:06:03 DEBUG "0.25942772836791833" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690363" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected 200) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1686690363, 22:06:03 DEBUG "0" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:18 INFO 22:06:18 INFO [loop_until]: kubectl --namespace=xlou top pods 22:06:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:06:18 INFO [loop_until]: OK (rc = 0) 22:06:18 DEBUG --- stdout --- 22:06:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 424Mi ds-cts-2 8m 393Mi ds-idrepo-0 3470m 13807Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3102m 4888Mi idm-58dc667486-rgz9j 2594m 5093Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 226m 640Mi 22:06:18 DEBUG --- stderr --- 22:06:18 DEBUG 22:06:22 INFO 22:06:22 INFO [loop_until]: kubectl --namespace=xlou top node 22:06:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:06:22 INFO [loop_until]: OK (rc = 0) 22:06:22 DEBUG --- stdout --- 22:06:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3123m 19% 5836Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 71m 0% 6822Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2678m 16% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 436m 2% 3121Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 63m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13713Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3351m 21% 14523Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14590Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 295m 1% 1954Mi 3% 22:06:22 DEBUG --- stderr --- 22:06:22 DEBUG 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:07:02 DEBUG "pod": "ds-idrepo-2" 22:07:02 DEBUG }, 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG { 22:07:02 DEBUG 22:07:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "pod": "ds-idrepo-0" 22:07:02 DEBUG }, 22:07:02 DEBUG "data": { 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "result": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0" 22:07:02 DEBUG { 22:07:02 DEBUG "637.8333333333334" 22:07:02 DEBUG ] 22:07:02 DEBUG "metric": { 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG "pod": "ds-cts-2" 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG }, 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG "value": [ 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 DEBUG 22:07:02 DEBUG } 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0.000018274258429905796" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "ds-cts-2" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0.2" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "ds-cts-0" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "ds-idrepo-0" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "637.8333333333334" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0.03333333333333333" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "ds-cts-0" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "ds-cts-2" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "ds-cts-0" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "ds-idrepo-0" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "instance": "10.142.0.100:9100" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "instance": "10.142.0.99:9100" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0.03333333333333333" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "13615804416" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "ds-cts-2" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "ds-idrepo-2" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "ds-idrepo-0" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "637.8333333333334" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "instance": "10.142.0.100:9100" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "instance": "10.142.0.99:9100" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "20276708.666666664" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "instance": "10.142.0.100:9100" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "instance": "10.142.0.99:9100" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "20274578.53333333" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "instance": "10.142.0.100:9100" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "instance": "10.142.0.99:9100" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "0.0001999999999990602" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690422" 22:07:02 INFO [http_cmd]: http status code OK 22:07:02 DEBUG --- status code --- 22:07:02 DEBUG http status code is 200 (expected 200) 22:07:02 DEBUG --- http response --- 22:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:02 DEBUG ----- output ----- 22:07:02 DEBUG { 22:07:02 DEBUG "status": "success", 22:07:02 DEBUG "data": { 22:07:02 DEBUG "resultType": "vector", 22:07:02 DEBUG "result": [ 22:07:02 DEBUG { 22:07:02 DEBUG "metric": { 22:07:02 DEBUG "pod": "ds-idrepo-2" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG [...] 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG 22:07:02 DEBUG "pod": "ds-idrepo-0" 22:07:02 DEBUG }, 22:07:02 DEBUG "value": [ 22:07:02 DEBUG 1686690422, 22:07:02 DEBUG "637.8333333333334" 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG ] 22:07:02 DEBUG } 22:07:02 DEBUG } 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690423" 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG http status code is 200 (expected 200) 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:03 DEBUG ----- output ----- 22:07:03 DEBUG { 22:07:03 DEBUG "status": "success", 22:07:03 DEBUG "data": { 22:07:03 DEBUG "resultType": "vector", 22:07:03 DEBUG "result": [ 22:07:03 DEBUG { 22:07:03 DEBUG "metric": { 22:07:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG [...] 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 1686690423, 22:07:03 DEBUG "0.2474505129087186" 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG } 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690423" 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG http status code is 200 (expected 200) 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:07:03 DEBUG ----- output ----- 22:07:03 DEBUG { 22:07:03 DEBUG "status": "success", 22:07:03 DEBUG "data": { 22:07:03 DEBUG "resultType": "vector", 22:07:03 DEBUG "result": [ 22:07:03 DEBUG { 22:07:03 DEBUG "metric": { 22:07:03 DEBUG "pod": "ds-cts-2" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG [...] 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 1686690423, 22:07:03 DEBUG "0" 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG } 22:07:18 INFO 22:07:18 INFO [loop_until]: kubectl --namespace=xlou top pods 22:07:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:07:18 INFO [loop_until]: OK (rc = 0) 22:07:18 DEBUG --- stdout --- 22:07:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 424Mi ds-cts-2 8m 392Mi ds-idrepo-0 3481m 13823Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2835m 4890Mi idm-58dc667486-rgz9j 2923m 5092Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 231m 639Mi 22:07:18 DEBUG --- stderr --- 22:07:18 DEBUG 22:07:22 INFO 22:07:22 INFO [loop_until]: kubectl --namespace=xlou top node 22:07:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:07:22 INFO [loop_until]: OK (rc = 0) 22:07:22 DEBUG --- stdout --- 22:07:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2886m 18% 5834Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 72m 0% 6860Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6823Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 3090m 19% 6074Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 452m 2% 3144Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13716Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3366m 21% 14502Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14594Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 296m 1% 1956Mi 3% 22:07:22 DEBUG --- stderr --- 22:07:22 DEBUG 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 DEBUG --- status code --- 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:02 DEBUG { 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG { 22:08:02 DEBUG { 22:08:02 DEBUG "data": { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "data": { 22:08:02 DEBUG "result": [ 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG { 22:08:02 DEBUG "result": [ 22:08:02 DEBUG "result": [ 22:08:02 DEBUG "metric": { 22:08:02 DEBUG { 22:08:02 DEBUG { 22:08:02 DEBUG "pod": "ds-idrepo-2" 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG }, 22:08:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:08:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:08:02 DEBUG "value": [ 22:08:02 DEBUG }, 22:08:02 DEBUG }, 22:08:02 DEBUG 22:08:02 DEBUG "value": [ 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "pod": "ds-idrepo-0" 22:08:02 DEBUG 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG 22:08:02 DEBUG }, 22:08:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:08:02 DEBUG "value": [ 22:08:02 DEBUG }, 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG }, 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG "value": [ 22:08:02 DEBUG "590.2" 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG ] 22:08:02 DEBUG "0" 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG "0" 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG { 22:08:02 DEBUG ] 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG "data": { 22:08:02 DEBUG ] 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG } 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG } 22:08:02 DEBUG "result": [ 22:08:02 DEBUG } 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG { 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG "metric": { 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG }, 22:08:02 DEBUG { 22:08:02 DEBUG "value": [ 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG 22:08:02 DEBUG "data": { 22:08:02 DEBUG 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG 22:08:02 DEBUG "result": [ 22:08:02 DEBUG [...] 22:08:02 DEBUG { 22:08:02 DEBUG 22:08:02 DEBUG "metric": { 22:08:02 DEBUG 22:08:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:08:02 DEBUG 22:08:02 DEBUG }, 22:08:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:08:02 DEBUG "value": [ 22:08:02 DEBUG }, 22:08:02 DEBUG 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG 22:08:02 DEBUG "0" 22:08:02 DEBUG [...] 22:08:02 DEBUG ] 22:08:02 DEBUG 22:08:02 DEBUG } 22:08:02 DEBUG 22:08:02 DEBUG ] 22:08:02 DEBUG 22:08:02 DEBUG } 22:08:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:08:02 DEBUG } 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "0.2" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "pod": "ds-cts-0" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "pod": "ds-idrepo-0" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "590.2666666666667" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "0" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "pod": "ds-cts-0" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "pod": "ds-cts-2" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "0" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "pod": "ds-cts-0" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "pod": "ds-idrepo-0" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "0" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690482" 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG [...] 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:02 DEBUG 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:08:02 DEBUG { 22:08:02 DEBUG }, 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "value": [ 22:08:02 DEBUG "data": { 22:08:02 DEBUG 1686690482, 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "0" 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG "result": [ 22:08:02 DEBUG ] 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG { 22:08:02 DEBUG } 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG "metric": { 22:08:02 DEBUG ] 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:02 DEBUG "instance": "10.142.0.100:9100" 22:08:02 DEBUG } 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG }, 22:08:02 DEBUG } 22:08:02 DEBUG { 22:08:02 DEBUG "value": [ 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG 22:08:02 DEBUG "data": { 22:08:02 DEBUG 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG 22:08:02 DEBUG "result": [ 22:08:02 DEBUG [...] 22:08:02 DEBUG { 22:08:02 DEBUG 22:08:02 DEBUG "metric": { 22:08:02 DEBUG 22:08:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:08:02 DEBUG 22:08:02 DEBUG }, 22:08:02 DEBUG "instance": "10.142.0.99:9100" 22:08:02 DEBUG "value": [ 22:08:02 DEBUG }, 22:08:02 DEBUG 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG 22:08:02 DEBUG "0.00033333333333333365" 22:08:02 DEBUG [...] 22:08:02 DEBUG ] 22:08:02 DEBUG 22:08:02 DEBUG } 22:08:02 DEBUG 22:08:02 DEBUG ] 22:08:02 DEBUG 22:08:02 DEBUG } 22:08:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:08:02 DEBUG } 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "13616664576" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "pod": "ds-idrepo-0" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "6746.531917199523" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "0" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "pod": "ds-idrepo-2" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "pod": "ds-idrepo-0" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "590.2" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "instance": "10.142.0.100:9100" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "instance": "10.142.0.99:9100" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "20262648.099999998" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "instance": "10.142.0.100:9100" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "instance": "10.142.0.99:9100" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "20262413.1" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690482" 22:08:02 INFO [http_cmd]: http status code OK 22:08:02 DEBUG --- status code --- 22:08:02 DEBUG http status code is 200 (expected 200) 22:08:02 DEBUG --- http response --- 22:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:02 DEBUG ----- output ----- 22:08:02 DEBUG { 22:08:02 DEBUG "status": "success", 22:08:02 DEBUG "data": { 22:08:02 DEBUG "resultType": "vector", 22:08:02 DEBUG "result": [ 22:08:02 DEBUG { 22:08:02 DEBUG "metric": { 22:08:02 DEBUG "instance": "10.142.0.100:9100" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG [...] 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG 22:08:02 DEBUG "instance": "10.142.0.99:9100" 22:08:02 DEBUG }, 22:08:02 DEBUG "value": [ 22:08:02 DEBUG 1686690482, 22:08:02 DEBUG "0.00043333333333445503" 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG ] 22:08:02 DEBUG } 22:08:02 DEBUG } 22:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690483" 22:08:03 INFO [http_cmd]: http status code OK 22:08:03 DEBUG --- status code --- 22:08:03 DEBUG http status code is 200 (expected 200) 22:08:03 DEBUG --- http response --- 22:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:03 DEBUG ----- output ----- 22:08:03 DEBUG { 22:08:03 DEBUG "status": "success", 22:08:03 DEBUG "data": { 22:08:03 DEBUG "resultType": "vector", 22:08:03 DEBUG "result": [ 22:08:03 DEBUG { 22:08:03 DEBUG "metric": { 22:08:03 DEBUG "pod": "ds-idrepo-2" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG [...] 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG "pod": "ds-idrepo-0" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 1686690483, 22:08:03 DEBUG "590.2" 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG } 22:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690483" 22:08:03 INFO [http_cmd]: http status code OK 22:08:03 DEBUG --- status code --- 22:08:03 DEBUG http status code is 200 (expected 200) 22:08:03 DEBUG --- http response --- 22:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:03 DEBUG ----- output ----- 22:08:03 DEBUG { 22:08:03 DEBUG "status": "success", 22:08:03 DEBUG "data": { 22:08:03 DEBUG "resultType": "vector", 22:08:03 DEBUG "result": [ 22:08:03 DEBUG { 22:08:03 DEBUG "metric": { 22:08:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG [...] 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 1686690483, 22:08:03 DEBUG "0.2456605342859109" 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG } 22:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690483" 22:08:03 INFO [http_cmd]: http status code OK 22:08:03 DEBUG --- status code --- 22:08:03 DEBUG http status code is 200 (expected 200) 22:08:03 DEBUG --- http response --- 22:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:08:03 DEBUG ----- output ----- 22:08:03 DEBUG { 22:08:03 DEBUG "status": "success", 22:08:03 DEBUG "data": { 22:08:03 DEBUG "resultType": "vector", 22:08:03 DEBUG "result": [ 22:08:03 DEBUG { 22:08:03 DEBUG "metric": { 22:08:03 DEBUG "pod": "ds-cts-2" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG [...] 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 1686690483, 22:08:03 DEBUG "0" 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG } 22:08:18 INFO 22:08:18 INFO [loop_until]: kubectl --namespace=xlou top pods 22:08:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:08:18 INFO [loop_until]: OK (rc = 0) 22:08:18 DEBUG --- stdout --- 22:08:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 424Mi ds-cts-2 8m 392Mi ds-idrepo-0 3274m 13807Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3314m 4892Mi idm-58dc667486-rgz9j 2510m 5093Mi lodemon-6455d5ff7c-zzndq 4m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 226m 640Mi 22:08:18 DEBUG --- stderr --- 22:08:18 DEBUG 22:08:22 INFO 22:08:22 INFO [loop_until]: kubectl --namespace=xlou top node 22:08:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:08:22 INFO [loop_until]: OK (rc = 0) 22:08:22 DEBUG --- stdout --- 22:08:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3364m 21% 5838Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 69m 0% 6826Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2527m 15% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 442m 2% 3146Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 78m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1194Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13715Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3224m 20% 14504Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14596Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 283m 1% 1956Mi 3% 22:08:22 DEBUG --- stderr --- 22:08:22 DEBUG 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:02 DEBUG { 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG { 22:09:02 DEBUG { 22:09:02 DEBUG "data": { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "data": { 22:09:02 DEBUG "result": [ 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG { 22:09:02 DEBUG "result": [ 22:09:02 DEBUG "result": [ 22:09:02 DEBUG "metric": { 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG { 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG { 22:09:02 DEBUG "pod": "ds-idrepo-2" 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG "metric": { 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG "metric": { 22:09:02 DEBUG }, 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:09:02 DEBUG "value": [ 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG }, 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG }, 22:09:02 DEBUG 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:02 DEBUG "value": [ 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG { 22:09:02 DEBUG 22:09:02 DEBUG { 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "data": { 22:09:02 DEBUG [...] 22:09:02 DEBUG "data": { 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "result": [ 22:09:02 DEBUG 22:09:02 DEBUG "result": [ 22:09:02 DEBUG 22:09:02 DEBUG "pod": "ds-idrepo-0" 22:09:02 DEBUG { 22:09:02 DEBUG 22:09:02 DEBUG { 22:09:02 DEBUG 22:09:02 DEBUG }, 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "pod": "ds-idrepo-0" 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:09:02 DEBUG "value": [ 22:09:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:09:02 DEBUG }, 22:09:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:09:02 DEBUG }, 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG "638.9333333333333" 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG ] 22:09:02 DEBUG 22:09:02 DEBUG "0" 22:09:02 DEBUG 22:09:02 DEBUG "0" 22:09:02 DEBUG } 22:09:02 DEBUG 22:09:02 DEBUG ] 22:09:02 DEBUG 22:09:02 DEBUG ] 22:09:02 DEBUG ] 22:09:02 DEBUG 22:09:02 DEBUG } 22:09:02 DEBUG 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 DEBUG [...] 22:09:02 DEBUG ] 22:09:02 DEBUG [...] 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG 22:09:02 DEBUG } 22:09:02 DEBUG 22:09:02 DEBUG } 22:09:02 DEBUG 22:09:02 DEBUG } 22:09:02 DEBUG 22:09:02 DEBUG } 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:09:02 DEBUG "pod": "ds-idrepo-0" 22:09:02 DEBUG }, 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "0.2" 22:09:02 DEBUG "0" 22:09:02 DEBUG ] 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "pod": "ds-cts-0" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "pod": "ds-cts-2" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "0" 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "pod": "ds-cts-0" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "pod": "ds-idrepo-0" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "638.9333333333333" 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "0" 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "pod": "ds-cts-0" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "pod": "ds-idrepo-0" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "0" 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690542" 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690542" 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG "metric": { 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG }, 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG "value": [ 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:02 DEBUG 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG [...] 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG }, 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:09:02 DEBUG "value": [ 22:09:02 DEBUG "instance": "10.142.0.100:9100" 22:09:02 DEBUG }, 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG "0" 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG ] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG } 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG ] 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG } 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG } 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:09:02 DEBUG "instance": "10.142.0.99:9100" 22:09:02 DEBUG }, 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "13618892800" 22:09:02 DEBUG "0.0003333333333333262" 22:09:02 DEBUG ] 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "0" 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "pod": "idm-58dc667486-rgz9j" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "0" 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690542" 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "pod": "ds-idrepo-2" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "pod": "ds-idrepo-0" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "638.9333333333333" 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690542" 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "instance": "10.142.0.100:9100" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "instance": "10.142.0.99:9100" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "17667169.46666667" 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690542" 22:09:02 INFO [http_cmd]: http status code OK 22:09:02 DEBUG --- status code --- 22:09:02 DEBUG http status code is 200 (expected 200) 22:09:02 DEBUG --- http response --- 22:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:02 DEBUG ----- output ----- 22:09:02 DEBUG { 22:09:02 DEBUG "status": "success", 22:09:02 DEBUG "data": { 22:09:02 DEBUG "resultType": "vector", 22:09:02 DEBUG "result": [ 22:09:02 DEBUG { 22:09:02 DEBUG "metric": { 22:09:02 DEBUG "instance": "10.142.0.100:9100" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG [...] 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG 22:09:02 DEBUG "instance": "10.142.0.99:9100" 22:09:02 DEBUG }, 22:09:02 DEBUG "value": [ 22:09:02 DEBUG 1686690542, 22:09:02 DEBUG "17665298.166666664" 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG ] 22:09:02 DEBUG } 22:09:02 DEBUG } 22:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690543" 22:09:03 INFO [http_cmd]: http status code OK 22:09:03 DEBUG --- status code --- 22:09:03 DEBUG http status code is 200 (expected 200) 22:09:03 DEBUG --- http response --- 22:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:03 DEBUG ----- output ----- 22:09:03 DEBUG { 22:09:03 DEBUG "status": "success", 22:09:03 DEBUG "data": { 22:09:03 DEBUG "resultType": "vector", 22:09:03 DEBUG "result": [ 22:09:03 DEBUG { 22:09:03 DEBUG "metric": { 22:09:03 DEBUG "instance": "10.142.0.100:9100" 22:09:03 DEBUG }, 22:09:03 DEBUG "value": [ 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG [...] 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG "instance": "10.142.0.99:9100" 22:09:03 DEBUG }, 22:09:03 DEBUG "value": [ 22:09:03 DEBUG 1686690543, 22:09:03 DEBUG "0.0006333333333335152" 22:09:03 DEBUG ] 22:09:03 DEBUG } 22:09:03 DEBUG ] 22:09:03 DEBUG } 22:09:03 DEBUG } 22:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690543" 22:09:03 INFO [http_cmd]: http status code OK 22:09:03 DEBUG --- status code --- 22:09:03 DEBUG http status code is 200 (expected 200) 22:09:03 DEBUG --- http response --- 22:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:03 DEBUG ----- output ----- 22:09:03 DEBUG { 22:09:03 DEBUG "status": "success", 22:09:03 DEBUG "data": { 22:09:03 DEBUG "resultType": "vector", 22:09:03 DEBUG "result": [ 22:09:03 DEBUG { 22:09:03 DEBUG "metric": { 22:09:03 DEBUG "pod": "ds-idrepo-2" 22:09:03 DEBUG }, 22:09:03 DEBUG "value": [ 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG [...] 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG "pod": "ds-idrepo-0" 22:09:03 DEBUG }, 22:09:03 DEBUG "value": [ 22:09:03 DEBUG 1686690543, 22:09:03 DEBUG "638.9333333333333" 22:09:03 DEBUG ] 22:09:03 DEBUG } 22:09:03 DEBUG ] 22:09:03 DEBUG } 22:09:03 DEBUG } 22:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690543" 22:09:03 INFO [http_cmd]: http status code OK 22:09:03 DEBUG --- status code --- 22:09:03 DEBUG http status code is 200 (expected 200) 22:09:03 DEBUG --- http response --- 22:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:03 DEBUG ----- output ----- 22:09:03 DEBUG { 22:09:03 DEBUG "status": "success", 22:09:03 DEBUG "data": { 22:09:03 DEBUG "resultType": "vector", 22:09:03 DEBUG "result": [ 22:09:03 DEBUG { 22:09:03 DEBUG "metric": { 22:09:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:09:03 DEBUG }, 22:09:03 DEBUG "value": [ 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG [...] 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:09:03 DEBUG }, 22:09:03 DEBUG "value": [ 22:09:03 DEBUG 1686690543, 22:09:03 DEBUG "0.22632954947458242" 22:09:03 DEBUG ] 22:09:03 DEBUG } 22:09:03 DEBUG ] 22:09:03 DEBUG } 22:09:03 DEBUG } 22:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690543" 22:09:03 INFO [http_cmd]: http status code OK 22:09:03 DEBUG --- status code --- 22:09:03 DEBUG http status code is 200 (expected 200) 22:09:03 DEBUG --- http response --- 22:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:09:03 DEBUG ----- output ----- 22:09:03 DEBUG { 22:09:03 DEBUG "status": "success", 22:09:03 DEBUG "data": { 22:09:03 DEBUG "resultType": "vector", 22:09:03 DEBUG "result": [ 22:09:03 DEBUG { 22:09:03 DEBUG "metric": { 22:09:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:09:03 DEBUG }, 22:09:03 DEBUG "value": [ 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG [...] 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG 22:09:03 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:09:03 DEBUG }, 22:09:03 DEBUG "value": [ 22:09:03 DEBUG 1686690543, 22:09:03 DEBUG "0" 22:09:03 DEBUG ] 22:09:03 DEBUG } 22:09:03 DEBUG ] 22:09:03 DEBUG } 22:09:03 DEBUG } 22:09:18 INFO 22:09:18 INFO [loop_until]: kubectl --namespace=xlou top pods 22:09:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:09:19 INFO [loop_until]: OK (rc = 0) 22:09:19 DEBUG --- stdout --- 22:09:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 9m 392Mi ds-idrepo-0 3589m 13807Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3680m 4894Mi idm-58dc667486-rgz9j 2955m 5094Mi lodemon-6455d5ff7c-zzndq 2m 69Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 223m 640Mi 22:09:19 DEBUG --- stderr --- 22:09:19 DEBUG 22:09:22 INFO 22:09:22 INFO [loop_until]: kubectl --namespace=xlou top node 22:09:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:09:22 INFO [loop_until]: OK (rc = 0) 22:09:22 DEBUG --- stdout --- 22:09:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3730m 23% 5837Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6828Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2923m 18% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 450m 2% 3129Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1196Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13717Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3844m 24% 14523Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14595Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 285m 1% 1958Mi 3% 22:09:22 DEBUG --- stderr --- 22:09:22 DEBUG 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "0.2" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "ds-idrepo-2" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "ds-idrepo-0" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "588.5" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "ds-cts-2" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "0" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "ds-cts-2" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "0" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "0" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "ds-cts-0" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "ds-idrepo-0" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "588.5" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "ds-cts-0" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "ds-cts-2" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "0" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:10:02 DEBUG "pod": "ds-cts-0" 22:10:02 DEBUG }, 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:10:02 DEBUG "pod": "ds-idrepo-0" 22:10:02 DEBUG }, 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "0" 22:10:02 DEBUG "0" 22:10:02 DEBUG ] 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690602" 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690602" 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG "metric": { 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG }, 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG "value": [ 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:02 DEBUG 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG 22:10:02 DEBUG { 22:10:02 DEBUG 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG [...] 22:10:02 DEBUG "data": { 22:10:02 DEBUG 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG "result": [ 22:10:02 DEBUG 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG { 22:10:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG "metric": { 22:10:02 DEBUG }, 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:10:02 DEBUG "value": [ 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:02 DEBUG }, 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG "value": [ 22:10:02 DEBUG "0" 22:10:02 DEBUG { 22:10:02 DEBUG 22:10:02 DEBUG ] 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG 22:10:02 DEBUG } 22:10:02 DEBUG "data": { 22:10:02 DEBUG 22:10:02 DEBUG ] 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG [...] 22:10:02 DEBUG } 22:10:02 DEBUG "result": [ 22:10:02 DEBUG 22:10:02 DEBUG } 22:10:02 DEBUG { 22:10:02 DEBUG 22:10:02 DEBUG "metric": { 22:10:02 DEBUG 22:10:02 DEBUG "instance": "10.142.0.100:9100" 22:10:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:10:02 DEBUG }, 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG 22:10:02 DEBUG "13617094656" 22:10:02 DEBUG 22:10:02 DEBUG ] 22:10:02 DEBUG [...] 22:10:02 DEBUG } 22:10:02 DEBUG 22:10:02 DEBUG ] 22:10:02 DEBUG 22:10:02 DEBUG } 22:10:02 DEBUG 22:10:02 DEBUG } 22:10:02 DEBUG "instance": "10.142.0.99:9100" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "0" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "0" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "ds-cts-2" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "0" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690602" 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "pod": "ds-idrepo-2" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "pod": "ds-idrepo-0" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "588.5" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690602" 22:10:02 INFO [http_cmd]: http status code OK 22:10:02 DEBUG --- status code --- 22:10:02 DEBUG http status code is 200 (expected 200) 22:10:02 DEBUG --- http response --- 22:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:02 DEBUG ----- output ----- 22:10:02 DEBUG { 22:10:02 DEBUG "status": "success", 22:10:02 DEBUG "data": { 22:10:02 DEBUG "resultType": "vector", 22:10:02 DEBUG "result": [ 22:10:02 DEBUG { 22:10:02 DEBUG "metric": { 22:10:02 DEBUG "instance": "10.142.0.100:9100" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG [...] 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG 22:10:02 DEBUG "instance": "10.142.0.99:9100" 22:10:02 DEBUG }, 22:10:02 DEBUG "value": [ 22:10:02 DEBUG 1686690602, 22:10:02 DEBUG "20140636.333333332" 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG ] 22:10:02 DEBUG } 22:10:02 DEBUG } 22:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690603" 22:10:03 INFO [http_cmd]: http status code OK 22:10:03 DEBUG --- status code --- 22:10:03 DEBUG http status code is 200 (expected 200) 22:10:03 DEBUG --- http response --- 22:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:03 DEBUG ----- output ----- 22:10:03 DEBUG { 22:10:03 DEBUG "status": "success", 22:10:03 DEBUG "data": { 22:10:03 DEBUG "resultType": "vector", 22:10:03 DEBUG "result": [ 22:10:03 DEBUG { 22:10:03 DEBUG "metric": { 22:10:03 DEBUG "instance": "10.142.0.100:9100" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG [...] 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG "instance": "10.142.0.99:9100" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 1686690603, 22:10:03 DEBUG "20336337" 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG } 22:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690603" 22:10:03 INFO [http_cmd]: http status code OK 22:10:03 DEBUG --- status code --- 22:10:03 DEBUG http status code is 200 (expected 200) 22:10:03 DEBUG --- http response --- 22:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:03 DEBUG ----- output ----- 22:10:03 DEBUG { 22:10:03 DEBUG "status": "success", 22:10:03 DEBUG "data": { 22:10:03 DEBUG "resultType": "vector", 22:10:03 DEBUG "result": [ 22:10:03 DEBUG { 22:10:03 DEBUG "metric": { 22:10:03 DEBUG "instance": "10.142.0.100:9100" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG [...] 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG "instance": "10.142.0.99:9100" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 1686690603, 22:10:03 DEBUG "0.0005666666666665302" 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG } 22:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690603" 22:10:03 INFO [http_cmd]: http status code OK 22:10:03 DEBUG --- status code --- 22:10:03 DEBUG http status code is 200 (expected 200) 22:10:03 DEBUG --- http response --- 22:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:03 DEBUG ----- output ----- 22:10:03 DEBUG { 22:10:03 DEBUG "status": "success", 22:10:03 DEBUG "data": { 22:10:03 DEBUG "resultType": "vector", 22:10:03 DEBUG "result": [ 22:10:03 DEBUG { 22:10:03 DEBUG "metric": { 22:10:03 DEBUG "pod": "ds-idrepo-2" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG [...] 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG "pod": "ds-idrepo-0" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 1686690603, 22:10:03 DEBUG "588.5333333333333" 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG } 22:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690603" 22:10:03 INFO [http_cmd]: http status code OK 22:10:03 DEBUG --- status code --- 22:10:03 DEBUG http status code is 200 (expected 200) 22:10:03 DEBUG --- http response --- 22:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:03 DEBUG ----- output ----- 22:10:03 DEBUG { 22:10:03 DEBUG "status": "success", 22:10:03 DEBUG "data": { 22:10:03 DEBUG "resultType": "vector", 22:10:03 DEBUG "result": [ 22:10:03 DEBUG { 22:10:03 DEBUG "metric": { 22:10:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG [...] 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 1686690603, 22:10:03 DEBUG "0.2390190091761939" 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG } 22:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690603" 22:10:03 INFO [http_cmd]: http status code OK 22:10:03 DEBUG --- status code --- 22:10:03 DEBUG http status code is 200 (expected 200) 22:10:03 DEBUG --- http response --- 22:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:10:03 DEBUG ----- output ----- 22:10:03 DEBUG { 22:10:03 DEBUG "status": "success", 22:10:03 DEBUG "data": { 22:10:03 DEBUG "resultType": "vector", 22:10:03 DEBUG "result": [ 22:10:03 DEBUG { 22:10:03 DEBUG "metric": { 22:10:03 DEBUG "pod": "ds-cts-2" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG [...] 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG 22:10:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:10:03 DEBUG }, 22:10:03 DEBUG "value": [ 22:10:03 DEBUG 1686690603, 22:10:03 DEBUG "0" 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG ] 22:10:03 DEBUG } 22:10:03 DEBUG } 22:10:19 INFO 22:10:19 INFO [loop_until]: kubectl --namespace=xlou top pods 22:10:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:10:19 INFO [loop_until]: OK (rc = 0) 22:10:19 DEBUG --- stdout --- 22:10:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 425Mi ds-cts-2 8m 392Mi ds-idrepo-0 3231m 13822Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3013m 4896Mi idm-58dc667486-rgz9j 2370m 5093Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 206m 640Mi 22:10:19 DEBUG --- stderr --- 22:10:19 DEBUG 22:10:22 INFO 22:10:22 INFO [loop_until]: kubectl --namespace=xlou top node 22:10:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:10:22 INFO [loop_until]: OK (rc = 0) 22:10:22 DEBUG --- stdout --- 22:10:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3219m 20% 5841Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6857Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6827Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2534m 15% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 436m 2% 3133Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 77m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 63m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1201Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13713Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3302m 20% 14521Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14593Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 274m 1% 1957Mi 3% 22:10:22 DEBUG --- stderr --- 22:10:22 DEBUG 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:11:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:11:02 DEBUG }, 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:11:02 DEBUG "pod": "idm-58dc667486-rgz9j" 22:11:02 DEBUG }, 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "0.2" 22:11:02 DEBUG "13849.83240819654" 22:11:02 DEBUG ] 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "idm-58dc667486-rgz9j" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "1.7842961041486718" 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "ds-idrepo-2" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "ds-idrepo-0" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "592.3666666666667" 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "0" 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "ds-cts-0" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "ds-idrepo-0" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "592.3666666666667" 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "ds-cts-0" 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "ds-cts-2" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "0" 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:11:02 DEBUG "pod": "ds-cts-0" 22:11:02 DEBUG }, 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:11:02 DEBUG "pod": "ds-idrepo-0" 22:11:02 DEBUG }, 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "0" 22:11:02 DEBUG "0" 22:11:02 DEBUG ] 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690662" 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "instance": "10.142.0.100:9100" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "instance": "10.142.0.99:9100" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "0" 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690662" 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG { 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG "data": { 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "data": { 22:11:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG }, 22:11:02 DEBUG "result": [ 22:11:02 DEBUG "value": [ 22:11:02 DEBUG { 22:11:02 DEBUG 22:11:02 DEBUG "metric": { 22:11:02 DEBUG 22:11:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:11:02 DEBUG 22:11:02 DEBUG }, 22:11:02 DEBUG [...] 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:11:02 DEBUG [...] 22:11:02 DEBUG }, 22:11:02 DEBUG 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG 22:11:02 DEBUG "0" 22:11:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:11:02 DEBUG ] 22:11:02 DEBUG }, 22:11:02 DEBUG } 22:11:02 DEBUG "value": [ 22:11:02 DEBUG ] 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG } 22:11:02 DEBUG "13616377856" 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "idm-58dc667486-rgz9j" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "0" 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "0" 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690662" 22:11:02 INFO [http_cmd]: http status code OK 22:11:02 DEBUG --- status code --- 22:11:02 DEBUG http status code is 200 (expected 200) 22:11:02 DEBUG --- http response --- 22:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:02 DEBUG ----- output ----- 22:11:02 DEBUG { 22:11:02 DEBUG "status": "success", 22:11:02 DEBUG "data": { 22:11:02 DEBUG "resultType": "vector", 22:11:02 DEBUG "result": [ 22:11:02 DEBUG { 22:11:02 DEBUG "metric": { 22:11:02 DEBUG "pod": "ds-idrepo-2" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG [...] 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG 22:11:02 DEBUG "pod": "ds-idrepo-0" 22:11:02 DEBUG }, 22:11:02 DEBUG "value": [ 22:11:02 DEBUG 1686690662, 22:11:02 DEBUG "592.3333333333334" 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG ] 22:11:02 DEBUG } 22:11:02 DEBUG } 22:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690663" 22:11:03 INFO [http_cmd]: http status code OK 22:11:03 DEBUG --- status code --- 22:11:03 DEBUG http status code is 200 (expected 200) 22:11:03 DEBUG --- http response --- 22:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:03 DEBUG ----- output ----- 22:11:03 DEBUG { 22:11:03 DEBUG "status": "success", 22:11:03 DEBUG "data": { 22:11:03 DEBUG "resultType": "vector", 22:11:03 DEBUG "result": [ 22:11:03 DEBUG { 22:11:03 DEBUG "metric": { 22:11:03 DEBUG "instance": "10.142.0.100:9100" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG [...] 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG "instance": "10.142.0.99:9100" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 1686690663, 22:11:03 DEBUG "19424938.53333333" 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG } 22:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690663" 22:11:03 INFO [http_cmd]: http status code OK 22:11:03 DEBUG --- status code --- 22:11:03 DEBUG http status code is 200 (expected 200) 22:11:03 DEBUG --- http response --- 22:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:03 DEBUG ----- output ----- 22:11:03 DEBUG { 22:11:03 DEBUG "status": "success", 22:11:03 DEBUG "data": { 22:11:03 DEBUG "resultType": "vector", 22:11:03 DEBUG "result": [ 22:11:03 DEBUG { 22:11:03 DEBUG "metric": { 22:11:03 DEBUG "instance": "10.142.0.100:9100" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG [...] 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG "instance": "10.142.0.99:9100" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 1686690663, 22:11:03 DEBUG "19421097.366666667" 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG } 22:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690663" 22:11:03 INFO [http_cmd]: http status code OK 22:11:03 DEBUG --- status code --- 22:11:03 DEBUG http status code is 200 (expected 200) 22:11:03 DEBUG --- http response --- 22:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:03 DEBUG ----- output ----- 22:11:03 DEBUG { 22:11:03 DEBUG "status": "success", 22:11:03 DEBUG "data": { 22:11:03 DEBUG "resultType": "vector", 22:11:03 DEBUG "result": [ 22:11:03 DEBUG { 22:11:03 DEBUG "metric": { 22:11:03 DEBUG "instance": "10.142.0.100:9100" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG [...] 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG "instance": "10.142.0.99:9100" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 1686690663, 22:11:03 DEBUG "0.00046666666666510536" 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG } 22:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690663" 22:11:03 INFO [http_cmd]: http status code OK 22:11:03 DEBUG --- status code --- 22:11:03 DEBUG http status code is 200 (expected 200) 22:11:03 DEBUG --- http response --- 22:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:03 DEBUG ----- output ----- 22:11:03 DEBUG { 22:11:03 DEBUG "status": "success", 22:11:03 DEBUG "data": { 22:11:03 DEBUG "resultType": "vector", 22:11:03 DEBUG "result": [ 22:11:03 DEBUG { 22:11:03 DEBUG "metric": { 22:11:03 DEBUG "pod": "ds-idrepo-2" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG [...] 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG "pod": "ds-idrepo-0" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 1686690663, 22:11:03 DEBUG "592.3333333333334" 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG } 22:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690663" 22:11:03 INFO [http_cmd]: http status code OK 22:11:03 DEBUG --- status code --- 22:11:03 DEBUG http status code is 200 (expected 200) 22:11:03 DEBUG --- http response --- 22:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:03 DEBUG ----- output ----- 22:11:03 DEBUG { 22:11:03 DEBUG "status": "success", 22:11:03 DEBUG "data": { 22:11:03 DEBUG "resultType": "vector", 22:11:03 DEBUG "result": [ 22:11:03 DEBUG { 22:11:03 DEBUG "metric": { 22:11:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG [...] 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 1686690663, 22:11:03 DEBUG "0.23715845500202862" 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG } 22:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690663" 22:11:03 INFO [http_cmd]: http status code OK 22:11:03 DEBUG --- status code --- 22:11:03 DEBUG http status code is 200 (expected 200) 22:11:03 DEBUG --- http response --- 22:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:03 DEBUG ----- output ----- 22:11:03 DEBUG { 22:11:03 DEBUG "status": "success", 22:11:03 DEBUG "data": { 22:11:03 DEBUG "resultType": "vector", 22:11:03 DEBUG "result": [ 22:11:03 DEBUG { 22:11:03 DEBUG "metric": { 22:11:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG [...] 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG 22:11:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:11:03 DEBUG }, 22:11:03 DEBUG "value": [ 22:11:03 DEBUG 1686690663, 22:11:03 DEBUG "0" 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG ] 22:11:03 DEBUG } 22:11:03 DEBUG } 22:11:19 INFO 22:11:19 INFO [loop_until]: kubectl --namespace=xlou top pods 22:11:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:11:19 INFO [loop_until]: OK (rc = 0) 22:11:19 DEBUG --- stdout --- 22:11:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 10m 416Mi ds-cts-1 8m 425Mi ds-cts-2 8m 392Mi ds-idrepo-0 3390m 13800Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3136m 4898Mi idm-58dc667486-rgz9j 2641m 5094Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 207m 640Mi 22:11:19 DEBUG --- stderr --- 22:11:19 DEBUG 22:11:22 INFO 22:11:22 INFO [loop_until]: kubectl --namespace=xlou top node 22:11:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:11:22 INFO [loop_until]: OK (rc = 0) 22:11:22 DEBUG --- stdout --- 22:11:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2995m 18% 5841Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 71m 0% 6829Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2688m 16% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 443m 2% 3134Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1201Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13715Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3276m 20% 14500Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14595Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 268m 1% 1957Mi 3% 22:11:22 DEBUG --- stderr --- 22:11:22 DEBUG 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "pod": "ds-cts-2" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 DEBUG 22:12:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "0" 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG ] 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG } 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG ] 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG } 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:12:02 DEBUG } 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "pod": "ds-cts-2" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "0" 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:12:02 DEBUG "pod": "ds-idrepo-2" 22:12:02 DEBUG }, 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:12:02 DEBUG "pod": "ds-idrepo-0" 22:12:02 DEBUG }, 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "0.2" 22:12:02 DEBUG "583.9333333333333" 22:12:02 DEBUG ] 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "0" 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "pod": "ds-cts-0" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "pod": "ds-idrepo-0" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "583.9333333333333" 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "pod": "ds-cts-0" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "pod": "ds-cts-2" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "0" 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:12:02 DEBUG "pod": "ds-cts-0" 22:12:02 DEBUG }, 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:12:02 DEBUG "pod": "ds-idrepo-0" 22:12:02 DEBUG }, 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "0.03333333333333333" 22:12:02 DEBUG "0" 22:12:02 DEBUG ] 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690722" 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "instance": "10.142.0.100:9100" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "instance": "10.142.0.99:9100" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "0" 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "0.03333333333333333" 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690722" 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG { 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG "data": { 22:12:02 DEBUG { 22:12:02 DEBUG { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG "data": { 22:12:02 DEBUG "data": { 22:12:02 DEBUG { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "result": [ 22:12:02 DEBUG "result": [ 22:12:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:12:02 DEBUG { 22:12:02 DEBUG { 22:12:02 DEBUG }, 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "value": [ 22:12:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:12:02 DEBUG "pod": "ds-cts-2" 22:12:02 DEBUG 22:12:02 DEBUG }, 22:12:02 DEBUG }, 22:12:02 DEBUG 22:12:02 DEBUG "value": [ 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG }, 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "value": [ 22:12:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:12:02 DEBUG "pod": "ds-idrepo-2" 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG }, 22:12:02 DEBUG }, 22:12:02 DEBUG "0" 22:12:02 DEBUG "value": [ 22:12:02 DEBUG "value": [ 22:12:02 DEBUG ] 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG } 22:12:02 DEBUG "13617377280" 22:12:02 DEBUG "0" 22:12:02 DEBUG ] 22:12:02 DEBUG ] 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690722" 22:12:02 INFO [http_cmd]: http status code OK 22:12:02 DEBUG --- status code --- 22:12:02 DEBUG http status code is 200 (expected 200) 22:12:02 DEBUG --- http response --- 22:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:02 DEBUG ----- output ----- 22:12:02 DEBUG { 22:12:02 DEBUG "status": "success", 22:12:02 DEBUG "data": { 22:12:02 DEBUG "resultType": "vector", 22:12:02 DEBUG "result": [ 22:12:02 DEBUG { 22:12:02 DEBUG "metric": { 22:12:02 DEBUG "pod": "ds-idrepo-2" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG [...] 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG 22:12:02 DEBUG "pod": "ds-idrepo-0" 22:12:02 DEBUG }, 22:12:02 DEBUG "value": [ 22:12:02 DEBUG 1686690722, 22:12:02 DEBUG "583.9666666666667" 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG ] 22:12:02 DEBUG } 22:12:02 DEBUG } 22:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690723" 22:12:03 INFO [http_cmd]: http status code OK 22:12:03 DEBUG --- status code --- 22:12:03 DEBUG http status code is 200 (expected 200) 22:12:03 DEBUG --- http response --- 22:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:03 DEBUG ----- output ----- 22:12:03 DEBUG { 22:12:03 DEBUG "status": "success", 22:12:03 DEBUG "data": { 22:12:03 DEBUG "resultType": "vector", 22:12:03 DEBUG "result": [ 22:12:03 DEBUG { 22:12:03 DEBUG "metric": { 22:12:03 DEBUG "instance": "10.142.0.100:9100" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG [...] 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG "instance": "10.142.0.99:9100" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 1686690723, 22:12:03 DEBUG "19092615.433333334" 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG } 22:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690723" 22:12:03 INFO [http_cmd]: http status code OK 22:12:03 DEBUG --- status code --- 22:12:03 DEBUG http status code is 200 (expected 200) 22:12:03 DEBUG --- http response --- 22:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:03 DEBUG ----- output ----- 22:12:03 DEBUG { 22:12:03 DEBUG "status": "success", 22:12:03 DEBUG "data": { 22:12:03 DEBUG "resultType": "vector", 22:12:03 DEBUG "result": [ 22:12:03 DEBUG { 22:12:03 DEBUG "metric": { 22:12:03 DEBUG "instance": "10.142.0.100:9100" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG [...] 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG "instance": "10.142.0.99:9100" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 1686690723, 22:12:03 DEBUG "19089408.633333333" 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG } 22:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690723" 22:12:03 INFO [http_cmd]: http status code OK 22:12:03 DEBUG --- status code --- 22:12:03 DEBUG http status code is 200 (expected 200) 22:12:03 DEBUG --- http response --- 22:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:03 DEBUG ----- output ----- 22:12:03 DEBUG { 22:12:03 DEBUG "status": "success", 22:12:03 DEBUG "data": { 22:12:03 DEBUG "resultType": "vector", 22:12:03 DEBUG "result": [ 22:12:03 DEBUG { 22:12:03 DEBUG "metric": { 22:12:03 DEBUG "instance": "10.142.0.100:9100" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG [...] 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG "instance": "10.142.0.99:9100" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 1686690723, 22:12:03 DEBUG "0.00050000000000144" 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG } 22:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690723" 22:12:03 INFO [http_cmd]: http status code OK 22:12:03 DEBUG --- status code --- 22:12:03 DEBUG http status code is 200 (expected 200) 22:12:03 DEBUG --- http response --- 22:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:03 DEBUG ----- output ----- 22:12:03 DEBUG { 22:12:03 DEBUG "status": "success", 22:12:03 DEBUG "data": { 22:12:03 DEBUG "resultType": "vector", 22:12:03 DEBUG "result": [ 22:12:03 DEBUG { 22:12:03 DEBUG "metric": { 22:12:03 DEBUG "pod": "ds-idrepo-2" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG [...] 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG "pod": "ds-idrepo-0" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 1686690723, 22:12:03 DEBUG "583.9666666666667" 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG } 22:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690723" 22:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690723" 22:12:03 INFO [http_cmd]: http status code OK 22:12:03 DEBUG --- status code --- 22:12:03 DEBUG http status code is 200 (expected 200) 22:12:03 DEBUG --- http response --- 22:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:03 DEBUG ----- output ----- 22:12:03 DEBUG { 22:12:03 DEBUG "status": "success", 22:12:03 DEBUG "data": { 22:12:03 DEBUG "resultType": "vector", 22:12:03 DEBUG "result": [ 22:12:03 DEBUG { 22:12:03 DEBUG "metric": { 22:12:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG [...] 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 1686690723, 22:12:03 DEBUG "0.23724563279859795" 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG } 22:12:03 INFO [http_cmd]: http status code OK 22:12:03 DEBUG --- status code --- 22:12:03 DEBUG http status code is 200 (expected 200) 22:12:03 DEBUG --- http response --- 22:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:12:03 DEBUG ----- output ----- 22:12:03 DEBUG { 22:12:03 DEBUG "status": "success", 22:12:03 DEBUG "data": { 22:12:03 DEBUG "resultType": "vector", 22:12:03 DEBUG "result": [ 22:12:03 DEBUG { 22:12:03 DEBUG "metric": { 22:12:03 DEBUG "pod": "ds-cts-2" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG [...] 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG 22:12:03 DEBUG "pod": "ds-idrepo-2" 22:12:03 DEBUG }, 22:12:03 DEBUG "value": [ 22:12:03 DEBUG 1686690723, 22:12:03 DEBUG "0.6925861797575948" 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG ] 22:12:03 DEBUG } 22:12:03 DEBUG } 22:12:19 INFO 22:12:19 INFO [loop_until]: kubectl --namespace=xlou top pods 22:12:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:12:19 INFO [loop_until]: OK (rc = 0) 22:12:19 DEBUG --- stdout --- 22:12:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 10m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3217m 13822Mi ds-idrepo-1 17m 13821Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2846m 4900Mi idm-58dc667486-rgz9j 2711m 5094Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 207m 640Mi 22:12:19 DEBUG --- stderr --- 22:12:19 DEBUG 22:12:22 INFO 22:12:22 INFO [loop_until]: kubectl --namespace=xlou top node 22:12:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:12:22 INFO [loop_until]: OK (rc = 0) 22:12:22 DEBUG --- stdout --- 22:12:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2949m 18% 5846Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 72m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2647m 16% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 443m 2% 3132Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13716Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3347m 21% 14501Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14596Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 273m 1% 1960Mi 3% 22:12:22 DEBUG --- stderr --- 22:12:22 DEBUG 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG { 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "result": [ 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG { 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "ds-idrepo-2" 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:13:02 DEBUG }, 22:13:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:13:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG }, 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG "value": [ 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "pod": "ds-idrepo-0" 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:13:02 DEBUG }, 22:13:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:13:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG }, 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "581.7333333333333" 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "312.75245046923874" 22:13:02 DEBUG ] 22:13:02 DEBUG "0.004762433638155774" 22:13:02 DEBUG "0.2" 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG ] 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG ] 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "0" 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "ds-cts-0" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "pod": "ds-idrepo-0" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "0" 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "ds-cts-0" 22:13:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:13:02 DEBUG }, 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "pod": "ds-idrepo-0" 22:13:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:13:02 DEBUG }, 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "581.7333333333333" 22:13:02 DEBUG "0" 22:13:02 DEBUG ] 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "ds-cts-0" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "pod": "ds-cts-2" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "0" 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690782" 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "instance": "10.142.0.100:9100" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "instance": "10.142.0.99:9100" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "0.0003333333333333854" 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "0" 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690782" 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG }, 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG "value": [ 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:02 DEBUG 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG [...] 22:13:02 DEBUG { 22:13:02 DEBUG 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG 22:13:02 DEBUG "data": { 22:13:02 DEBUG 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:13:02 DEBUG "result": [ 22:13:02 DEBUG }, 22:13:02 DEBUG { 22:13:02 DEBUG "value": [ 22:13:02 DEBUG "metric": { 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:13:02 DEBUG "0" 22:13:02 DEBUG }, 22:13:02 DEBUG ] 22:13:02 DEBUG "value": [ 22:13:02 DEBUG } 22:13:02 DEBUG 22:13:02 DEBUG ] 22:13:02 DEBUG 22:13:02 DEBUG } 22:13:02 DEBUG 22:13:02 DEBUG } 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "13616549888" 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "0" 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690782" 22:13:02 INFO [http_cmd]: http status code OK 22:13:02 DEBUG --- status code --- 22:13:02 DEBUG http status code is 200 (expected 200) 22:13:02 DEBUG --- http response --- 22:13:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:02 DEBUG ----- output ----- 22:13:02 DEBUG { 22:13:02 DEBUG "status": "success", 22:13:02 DEBUG "data": { 22:13:02 DEBUG "resultType": "vector", 22:13:02 DEBUG "result": [ 22:13:02 DEBUG { 22:13:02 DEBUG "metric": { 22:13:02 DEBUG "pod": "ds-idrepo-2" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG [...] 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG 22:13:02 DEBUG "pod": "ds-idrepo-0" 22:13:02 DEBUG }, 22:13:02 DEBUG "value": [ 22:13:02 DEBUG 1686690782, 22:13:02 DEBUG "581.7333333333333" 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG ] 22:13:02 DEBUG } 22:13:02 DEBUG } 22:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690783" 22:13:03 INFO [http_cmd]: http status code OK 22:13:03 DEBUG --- status code --- 22:13:03 DEBUG http status code is 200 (expected 200) 22:13:03 DEBUG --- http response --- 22:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:03 DEBUG ----- output ----- 22:13:03 DEBUG { 22:13:03 DEBUG "status": "success", 22:13:03 DEBUG "data": { 22:13:03 DEBUG "resultType": "vector", 22:13:03 DEBUG "result": [ 22:13:03 DEBUG { 22:13:03 DEBUG "metric": { 22:13:03 DEBUG "instance": "10.142.0.100:9100" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG [...] 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG "instance": "10.142.0.99:9100" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 1686690783, 22:13:03 DEBUG "19202388.8" 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG } 22:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690783" 22:13:03 INFO [http_cmd]: http status code OK 22:13:03 DEBUG --- status code --- 22:13:03 DEBUG http status code is 200 (expected 200) 22:13:03 DEBUG --- http response --- 22:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:03 DEBUG ----- output ----- 22:13:03 DEBUG { 22:13:03 DEBUG "status": "success", 22:13:03 DEBUG "data": { 22:13:03 DEBUG "resultType": "vector", 22:13:03 DEBUG "result": [ 22:13:03 DEBUG { 22:13:03 DEBUG "metric": { 22:13:03 DEBUG "instance": "10.142.0.100:9100" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG [...] 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG "instance": "10.142.0.99:9100" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 1686690783, 22:13:03 DEBUG "19198281.733333334" 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG } 22:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690783" 22:13:03 INFO [http_cmd]: http status code OK 22:13:03 DEBUG --- status code --- 22:13:03 DEBUG http status code is 200 (expected 200) 22:13:03 DEBUG --- http response --- 22:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:03 DEBUG ----- output ----- 22:13:03 DEBUG { 22:13:03 DEBUG "status": "success", 22:13:03 DEBUG "data": { 22:13:03 DEBUG "resultType": "vector", 22:13:03 DEBUG "result": [ 22:13:03 DEBUG { 22:13:03 DEBUG "metric": { 22:13:03 DEBUG "instance": "10.142.0.100:9100" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG [...] 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG "instance": "10.142.0.99:9100" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 1686690783, 22:13:03 DEBUG "0.0005666666666665302" 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG } 22:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690783" 22:13:03 INFO [http_cmd]: http status code OK 22:13:03 DEBUG --- status code --- 22:13:03 DEBUG http status code is 200 (expected 200) 22:13:03 DEBUG --- http response --- 22:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:03 DEBUG ----- output ----- 22:13:03 DEBUG { 22:13:03 DEBUG "status": "success", 22:13:03 DEBUG "data": { 22:13:03 DEBUG "resultType": "vector", 22:13:03 DEBUG "result": [ 22:13:03 DEBUG { 22:13:03 DEBUG "metric": { 22:13:03 DEBUG "pod": "ds-idrepo-2" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG [...] 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG "pod": "ds-idrepo-0" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 1686690783, 22:13:03 DEBUG "581.7333333333333" 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG } 22:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690783" 22:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690783" 22:13:03 INFO [http_cmd]: http status code OK 22:13:03 DEBUG --- status code --- 22:13:03 DEBUG http status code is 200 (expected 200) 22:13:03 DEBUG --- http response --- 22:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:03 DEBUG ----- output ----- 22:13:03 DEBUG { 22:13:03 DEBUG "status": "success", 22:13:03 DEBUG "data": { 22:13:03 DEBUG "resultType": "vector", 22:13:03 DEBUG "result": [ 22:13:03 DEBUG { 22:13:03 DEBUG "metric": { 22:13:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG [...] 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 1686690783, 22:13:03 DEBUG "0.23235706598064068" 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG } 22:13:03 INFO [http_cmd]: http status code OK 22:13:03 DEBUG --- status code --- 22:13:03 DEBUG http status code is 200 (expected 200) 22:13:03 DEBUG --- http response --- 22:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:03 DEBUG ----- output ----- 22:13:03 DEBUG { 22:13:03 DEBUG "status": "success", 22:13:03 DEBUG "data": { 22:13:03 DEBUG "resultType": "vector", 22:13:03 DEBUG "result": [ 22:13:03 DEBUG { 22:13:03 DEBUG "metric": { 22:13:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG [...] 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG 22:13:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:13:03 DEBUG }, 22:13:03 DEBUG "value": [ 22:13:03 DEBUG 1686690783, 22:13:03 DEBUG "0" 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG ] 22:13:03 DEBUG } 22:13:03 DEBUG } 22:13:19 INFO 22:13:19 INFO [loop_until]: kubectl --namespace=xlou top pods 22:13:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:13:19 INFO [loop_until]: OK (rc = 0) 22:13:19 DEBUG --- stdout --- 22:13:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 8m 416Mi ds-cts-1 9m 424Mi ds-cts-2 8m 392Mi ds-idrepo-0 3350m 13822Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 16m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2930m 4902Mi idm-58dc667486-rgz9j 2569m 5094Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 199m 640Mi 22:13:19 DEBUG --- stderr --- 22:13:19 DEBUG 22:13:22 INFO 22:13:22 INFO [loop_until]: kubectl --namespace=xlou top node 22:13:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:13:22 INFO [loop_until]: OK (rc = 0) 22:13:22 DEBUG --- stdout --- 22:13:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2985m 18% 5846Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 72m 0% 6855Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6826Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2693m 16% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6952Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 445m 2% 3124Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 59m 0% 1193Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 77m 0% 13715Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3364m 21% 14501Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14591Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 272m 1% 1957Mi 3% 22:13:22 DEBUG --- stderr --- 22:13:22 DEBUG 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "pod": "ds-idrepo-2" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "ds-idrepo-0" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "564.8" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG ] 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG } 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG ] 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG } 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG } 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG { 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG "data": { 22:14:02 DEBUG { 22:14:02 DEBUG { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG "data": { 22:14:02 DEBUG "data": { 22:14:02 DEBUG { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "result": [ 22:14:02 DEBUG "result": [ 22:14:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:14:02 DEBUG { 22:14:02 DEBUG { 22:14:02 DEBUG }, 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "value": [ 22:14:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:14:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:14:02 DEBUG 22:14:02 DEBUG }, 22:14:02 DEBUG }, 22:14:02 DEBUG 22:14:02 DEBUG "value": [ 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG }, 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "value": [ 22:14:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:14:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG }, 22:14:02 DEBUG }, 22:14:02 DEBUG "0" 22:14:02 DEBUG "value": [ 22:14:02 DEBUG "value": [ 22:14:02 DEBUG ] 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG } 22:14:02 DEBUG "0" 22:14:02 DEBUG "0" 22:14:02 DEBUG ] 22:14:02 DEBUG ] 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG } 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "0.2" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "pod": "ds-cts-0" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "ds-idrepo-0" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "0" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "0" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "pod": "ds-cts-0" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "ds-idrepo-0" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "564.8" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "pod": "ds-cts-0" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "ds-cts-2" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "0" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690842" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "instance": "10.142.0.100:9100" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "instance": "10.142.0.99:9100" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "0" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "0" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "0" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690842" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "13615595520" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690842" 22:14:02 INFO [http_cmd]: http status code OK 22:14:02 DEBUG --- status code --- 22:14:02 DEBUG http status code is 200 (expected 200) 22:14:02 DEBUG --- http response --- 22:14:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:02 DEBUG ----- output ----- 22:14:02 DEBUG { 22:14:02 DEBUG "status": "success", 22:14:02 DEBUG "data": { 22:14:02 DEBUG "resultType": "vector", 22:14:02 DEBUG "result": [ 22:14:02 DEBUG { 22:14:02 DEBUG "metric": { 22:14:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG [...] 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG 22:14:02 DEBUG "pod": "lodemon-6455d5ff7c-zzndq" 22:14:02 DEBUG }, 22:14:02 DEBUG "value": [ 22:14:02 DEBUG 1686690842, 22:14:02 DEBUG "0" 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG ] 22:14:02 DEBUG } 22:14:02 DEBUG } 22:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690843" 22:14:03 INFO [http_cmd]: http status code OK 22:14:03 DEBUG --- status code --- 22:14:03 DEBUG http status code is 200 (expected 200) 22:14:03 DEBUG --- http response --- 22:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:03 DEBUG ----- output ----- 22:14:03 DEBUG { 22:14:03 DEBUG "status": "success", 22:14:03 DEBUG "data": { 22:14:03 DEBUG "resultType": "vector", 22:14:03 DEBUG "result": [ 22:14:03 DEBUG { 22:14:03 DEBUG "metric": { 22:14:03 DEBUG "pod": "ds-idrepo-2" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG [...] 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG "pod": "ds-idrepo-0" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 1686690843, 22:14:03 DEBUG "564.8" 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG } 22:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690843" 22:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690843" 22:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690843" 22:14:03 INFO [http_cmd]: http status code OK 22:14:03 INFO [http_cmd]: http status code OK 22:14:03 DEBUG --- status code --- 22:14:03 DEBUG --- status code --- 22:14:03 DEBUG http status code is 200 (expected 200) 22:14:03 DEBUG http status code is 200 (expected 200) 22:14:03 DEBUG --- http response --- 22:14:03 DEBUG --- http response --- 22:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:03 DEBUG ----- output ----- 22:14:03 DEBUG ----- output ----- 22:14:03 DEBUG { 22:14:03 DEBUG { 22:14:03 DEBUG "status": "success", 22:14:03 DEBUG "status": "success", 22:14:03 DEBUG "data": { 22:14:03 DEBUG "data": { 22:14:03 DEBUG "resultType": "vector", 22:14:03 DEBUG "resultType": "vector", 22:14:03 DEBUG "result": [ 22:14:03 DEBUG "result": [ 22:14:03 DEBUG { 22:14:03 DEBUG { 22:14:03 DEBUG "metric": { 22:14:03 DEBUG "metric": { 22:14:03 DEBUG "instance": "10.142.0.100:9100" 22:14:03 DEBUG "instance": "10.142.0.100:9100" 22:14:03 DEBUG }, 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG [...] 22:14:03 DEBUG [...] 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG "instance": "10.142.0.99:9100" 22:14:03 DEBUG "instance": "10.142.0.99:9100" 22:14:03 DEBUG }, 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 1686690843, 22:14:03 DEBUG 1686690843, 22:14:03 DEBUG "0.0006000000000009701" 22:14:03 DEBUG "20341617.333333336" 22:14:03 DEBUG ] 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG } 22:14:03 DEBUG ] 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG } 22:14:03 DEBUG } 22:14:03 DEBUG } 22:14:03 INFO [http_cmd]: http status code OK 22:14:03 DEBUG --- status code --- 22:14:03 DEBUG http status code is 200 (expected 200) 22:14:03 DEBUG --- http response --- 22:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:03 DEBUG ----- output ----- 22:14:03 DEBUG { 22:14:03 DEBUG "status": "success", 22:14:03 DEBUG "data": { 22:14:03 DEBUG "resultType": "vector", 22:14:03 DEBUG "result": [ 22:14:03 DEBUG { 22:14:03 DEBUG "metric": { 22:14:03 DEBUG "instance": "10.142.0.100:9100" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG [...] 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG "instance": "10.142.0.99:9100" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 1686690843, 22:14:03 DEBUG "20337803.299999997" 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG } 22:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690843" 22:14:03 INFO [http_cmd]: http status code OK 22:14:03 DEBUG --- status code --- 22:14:03 DEBUG http status code is 200 (expected 200) 22:14:03 DEBUG --- http response --- 22:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:03 DEBUG ----- output ----- 22:14:03 DEBUG { 22:14:03 DEBUG "status": "success", 22:14:03 DEBUG "data": { 22:14:03 DEBUG "resultType": "vector", 22:14:03 DEBUG "result": [ 22:14:03 DEBUG { 22:14:03 DEBUG "metric": { 22:14:03 DEBUG "pod": "ds-idrepo-2" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG [...] 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG "pod": "ds-idrepo-0" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 1686690843, 22:14:03 DEBUG "564.8" 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG } 22:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690843" 22:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690843" 22:14:03 INFO [http_cmd]: http status code OK 22:14:03 DEBUG --- status code --- 22:14:03 DEBUG http status code is 200 (expected 200) 22:14:03 DEBUG --- http response --- 22:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:03 DEBUG ----- output ----- 22:14:03 DEBUG { 22:14:03 DEBUG "status": "success", 22:14:03 DEBUG "data": { 22:14:03 DEBUG "resultType": "vector", 22:14:03 DEBUG "result": [ 22:14:03 DEBUG { 22:14:03 DEBUG "metric": { 22:14:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG [...] 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 1686690843, 22:14:03 DEBUG "0.23655873981489323" 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG } 22:14:03 INFO [http_cmd]: http status code OK 22:14:03 DEBUG --- status code --- 22:14:03 DEBUG http status code is 200 (expected 200) 22:14:03 DEBUG --- http response --- 22:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:14:03 DEBUG ----- output ----- 22:14:03 DEBUG { 22:14:03 DEBUG "status": "success", 22:14:03 DEBUG "data": { 22:14:03 DEBUG "resultType": "vector", 22:14:03 DEBUG "result": [ 22:14:03 DEBUG { 22:14:03 DEBUG "metric": { 22:14:03 DEBUG "pod": "ds-cts-2" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG [...] 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG 22:14:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:14:03 DEBUG }, 22:14:03 DEBUG "value": [ 22:14:03 DEBUG 1686690843, 22:14:03 DEBUG "0" 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG ] 22:14:03 DEBUG } 22:14:03 DEBUG } 22:14:19 INFO 22:14:19 INFO [loop_until]: kubectl --namespace=xlou top pods 22:14:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:14:19 INFO [loop_until]: OK (rc = 0) 22:14:19 DEBUG --- stdout --- 22:14:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 9m 392Mi ds-idrepo-0 3186m 13805Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3142m 4904Mi idm-58dc667486-rgz9j 2660m 5094Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 203m 640Mi 22:14:19 DEBUG --- stderr --- 22:14:19 DEBUG 22:14:23 INFO 22:14:23 INFO [loop_until]: kubectl --namespace=xlou top node 22:14:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:14:23 INFO [loop_until]: OK (rc = 0) 22:14:23 DEBUG --- stdout --- 22:14:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3235m 20% 5847Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 72m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6826Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2824m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 80m 0% 6962Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 442m 2% 3144Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 75m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 60m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13717Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3480m 21% 14523Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 74m 0% 14592Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 260m 1% 1968Mi 3% 22:14:23 DEBUG --- stderr --- 22:14:23 DEBUG 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "ds-idrepo-0" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "443234180.5100988" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "0" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "ds-idrepo-2" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "ds-idrepo-0" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "611.2666666666667" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "0.2" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "ds-idrepo-0" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "3.0723945672415063" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "ds-cts-0" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "ds-idrepo-0" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "0" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "0" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "ds-cts-0" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "ds-idrepo-0" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "611.3" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690902" 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "ds-cts-0" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "ds-cts-2" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "0" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "instance": "10.142.0.100:9100" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "instance": "10.142.0.99:9100" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "0" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690902" 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG "data": { 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG "result": [ 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG { 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:02 DEBUG "metric": { 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:15:02 DEBUG { 22:15:02 DEBUG }, 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "value": [ 22:15:02 DEBUG "data": { 22:15:02 DEBUG 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG 22:15:02 DEBUG "result": [ 22:15:02 DEBUG 22:15:02 DEBUG { 22:15:02 DEBUG [...] 22:15:02 DEBUG "metric": { 22:15:02 DEBUG 22:15:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:15:02 DEBUG 22:15:02 DEBUG }, 22:15:02 DEBUG 22:15:02 DEBUG "value": [ 22:15:02 DEBUG "pod": "ds-idrepo-0" 22:15:02 DEBUG 22:15:02 DEBUG }, 22:15:02 DEBUG 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG [...] 22:15:02 DEBUG "6312.457614624797" 22:15:02 DEBUG 22:15:02 DEBUG ] 22:15:02 DEBUG 22:15:02 DEBUG } 22:15:02 DEBUG 22:15:02 DEBUG ] 22:15:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:15:02 DEBUG } 22:15:02 DEBUG }, 22:15:02 DEBUG } 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "0" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:02 INFO [http_cmd]: http status code OK 22:15:02 DEBUG --- status code --- 22:15:02 DEBUG http status code is 200 (expected 200) 22:15:02 DEBUG --- http response --- 22:15:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:02 DEBUG ----- output ----- 22:15:02 DEBUG { 22:15:02 DEBUG "status": "success", 22:15:02 DEBUG "data": { 22:15:02 DEBUG "resultType": "vector", 22:15:02 DEBUG "result": [ 22:15:02 DEBUG { 22:15:02 DEBUG "metric": { 22:15:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG [...] 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG 22:15:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:15:02 DEBUG }, 22:15:02 DEBUG "value": [ 22:15:02 DEBUG 1686690902, 22:15:02 DEBUG "0" 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG ] 22:15:02 DEBUG } 22:15:02 DEBUG } 22:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690903" 22:15:03 INFO [http_cmd]: http status code OK 22:15:03 DEBUG --- status code --- 22:15:03 DEBUG http status code is 200 (expected 200) 22:15:03 DEBUG --- http response --- 22:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:03 DEBUG ----- output ----- 22:15:03 DEBUG { 22:15:03 DEBUG "status": "success", 22:15:03 DEBUG "data": { 22:15:03 DEBUG "resultType": "vector", 22:15:03 DEBUG "result": [ 22:15:03 DEBUG { 22:15:03 DEBUG "metric": { 22:15:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG [...] 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 1686690903, 22:15:03 DEBUG "13617836032" 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG } 22:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690903" 22:15:03 INFO [http_cmd]: http status code OK 22:15:03 DEBUG --- status code --- 22:15:03 DEBUG http status code is 200 (expected 200) 22:15:03 DEBUG --- http response --- 22:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:03 DEBUG ----- output ----- 22:15:03 DEBUG { 22:15:03 DEBUG "status": "success", 22:15:03 DEBUG "data": { 22:15:03 DEBUG "resultType": "vector", 22:15:03 DEBUG "result": [ 22:15:03 DEBUG { 22:15:03 DEBUG "metric": { 22:15:03 DEBUG "pod": "ds-idrepo-2" 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG [...] 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG "pod": "ds-idrepo-0" 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 1686690903, 22:15:03 DEBUG "611.2333333333333" 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG } 22:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690903" 22:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690903" 22:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690903" 22:15:03 INFO [http_cmd]: http status code OK 22:15:03 DEBUG --- status code --- 22:15:03 DEBUG http status code is 200 (expected 200) 22:15:03 DEBUG --- http response --- 22:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:03 DEBUG ----- output ----- 22:15:03 DEBUG { 22:15:03 DEBUG "status": "success", 22:15:03 DEBUG "data": { 22:15:03 DEBUG "resultType": "vector", 22:15:03 DEBUG "result": [ 22:15:03 DEBUG { 22:15:03 DEBUG "metric": { 22:15:03 DEBUG "instance": "10.142.0.100:9100" 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG [...] 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG "instance": "10.142.0.99:9100" 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 1686690903, 22:15:03 DEBUG "0.0002999999999985903" 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG } 22:15:03 INFO [http_cmd]: http status code OK 22:15:03 INFO [http_cmd]: http status code OK 22:15:03 DEBUG --- status code --- 22:15:03 DEBUG --- status code --- 22:15:03 DEBUG http status code is 200 (expected 200) 22:15:03 DEBUG http status code is 200 (expected 200) 22:15:03 DEBUG --- http response --- 22:15:03 DEBUG --- http response --- 22:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:03 DEBUG ----- output ----- 22:15:03 DEBUG ----- output ----- 22:15:03 DEBUG { 22:15:03 DEBUG { 22:15:03 DEBUG "status": "success", 22:15:03 DEBUG "status": "success", 22:15:03 DEBUG "data": { 22:15:03 DEBUG "data": { 22:15:03 DEBUG "resultType": "vector", 22:15:03 DEBUG "resultType": "vector", 22:15:03 DEBUG "result": [ 22:15:03 DEBUG "result": [ 22:15:03 DEBUG { 22:15:03 DEBUG { 22:15:03 DEBUG "metric": { 22:15:03 DEBUG "metric": { 22:15:03 DEBUG "instance": "10.142.0.100:9100" 22:15:03 DEBUG "instance": "10.142.0.100:9100" 22:15:03 DEBUG }, 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG [...] 22:15:03 DEBUG [...] 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG "instance": "10.142.0.99:9100" 22:15:03 DEBUG "instance": "10.142.0.99:9100" 22:15:03 DEBUG }, 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 1686690903, 22:15:03 DEBUG 1686690903, 22:15:03 DEBUG "20095951.933333334" 22:15:03 DEBUG "20092194.266666666" 22:15:03 DEBUG ] 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG } 22:15:03 DEBUG ] 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG } 22:15:03 DEBUG } 22:15:03 DEBUG } 22:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690903" 22:15:03 INFO [http_cmd]: http status code OK 22:15:03 DEBUG --- status code --- 22:15:03 DEBUG http status code is 200 (expected 200) 22:15:03 DEBUG --- http response --- 22:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:03 DEBUG ----- output ----- 22:15:03 DEBUG { 22:15:03 DEBUG "status": "success", 22:15:03 DEBUG "data": { 22:15:03 DEBUG "resultType": "vector", 22:15:03 DEBUG "result": [ 22:15:03 DEBUG { 22:15:03 DEBUG "metric": { 22:15:03 DEBUG "pod": "ds-idrepo-2" 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG [...] 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG "pod": "ds-idrepo-0" 22:15:03 DEBUG }, 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 1686690903, 22:15:03 DEBUG "611.2666666666667" 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG } 22:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690903" 22:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690903" 22:15:03 INFO [http_cmd]: http status code OK 22:15:03 DEBUG --- status code --- 22:15:03 DEBUG http status code is 200 (expected 200) 22:15:03 DEBUG --- http response --- 22:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:03 INFO [http_cmd]: http status code OK 22:15:03 DEBUG ----- output ----- 22:15:03 DEBUG --- status code --- 22:15:03 DEBUG { 22:15:03 DEBUG http status code is 200 (expected 200) 22:15:03 DEBUG "status": "success", 22:15:03 DEBUG --- http response --- 22:15:03 DEBUG "data": { 22:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:03 DEBUG "resultType": "vector", 22:15:03 DEBUG ----- output ----- 22:15:03 DEBUG "result": [ 22:15:03 DEBUG { 22:15:03 DEBUG { 22:15:03 DEBUG "status": "success", 22:15:03 DEBUG "metric": { 22:15:03 DEBUG "data": { 22:15:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:15:03 DEBUG "resultType": "vector", 22:15:03 DEBUG }, 22:15:03 DEBUG "result": [ 22:15:03 DEBUG "value": [ 22:15:03 DEBUG { 22:15:03 DEBUG 22:15:03 DEBUG "metric": { 22:15:03 DEBUG 22:15:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:15:03 DEBUG 22:15:03 DEBUG }, 22:15:03 DEBUG [...] 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG 22:15:03 DEBUG "pod": "ds-idrepo-0" 22:15:03 DEBUG [...] 22:15:03 DEBUG }, 22:15:03 DEBUG 22:15:03 DEBUG "value": [ 22:15:03 DEBUG 22:15:03 DEBUG 1686690903, 22:15:03 DEBUG 22:15:03 DEBUG "0.5602240896358543" 22:15:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:15:03 DEBUG ] 22:15:03 DEBUG }, 22:15:03 DEBUG } 22:15:03 DEBUG "value": [ 22:15:03 DEBUG ] 22:15:03 DEBUG 1686690903, 22:15:03 DEBUG } 22:15:03 DEBUG "0.22662591807729265" 22:15:03 DEBUG } 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG ] 22:15:03 DEBUG } 22:15:03 DEBUG } 22:15:19 INFO 22:15:19 INFO [loop_until]: kubectl --namespace=xlou top pods 22:15:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:15:19 INFO [loop_until]: OK (rc = 0) 22:15:19 DEBUG --- stdout --- 22:15:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 10m 5754Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3361m 13801Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 17m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3407m 4906Mi idm-58dc667486-rgz9j 2801m 5095Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 212m 640Mi 22:15:19 DEBUG --- stderr --- 22:15:19 DEBUG 22:15:23 INFO 22:15:23 INFO [loop_until]: kubectl --namespace=xlou top node 22:15:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:15:23 INFO [loop_until]: OK (rc = 0) 22:15:23 DEBUG --- stdout --- 22:15:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3113m 19% 5851Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6828Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2811m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 68m 0% 6952Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 425m 2% 3147Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 77m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 59m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1196Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13716Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3381m 21% 14523Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14592Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 279m 1% 1955Mi 3% 22:15:23 DEBUG --- stderr --- 22:15:23 DEBUG 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG { 22:16:02 DEBUG { 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "data": { 22:16:02 DEBUG "data": { 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "result": [ 22:16:02 DEBUG "result": [ 22:16:02 DEBUG { 22:16:02 DEBUG { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:16:02 DEBUG "pod": "ds-cts-2" 22:16:02 DEBUG }, 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG [...] 22:16:02 DEBUG [...] 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:16:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:16:02 DEBUG }, 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG "0" 22:16:02 DEBUG "0" 22:16:02 DEBUG ] 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 DEBUG ] 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG { 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "data": { 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "result": [ 22:16:02 DEBUG { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG [...] 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG "0.2" 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG { 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "data": { 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "result": [ 22:16:02 DEBUG { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "pod": "ds-idrepo-2" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG [...] 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG "pod": "ds-idrepo-0" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG "639.4" 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG { 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "data": { 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "result": [ 22:16:02 DEBUG { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "pod": "ds-cts-2" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG [...] 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG "0" 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG { 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "data": { 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "result": [ 22:16:02 DEBUG { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "pod": "ds-cts-0" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG [...] 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG "pod": "ds-idrepo-0" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG "0" 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690962" 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG { 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "data": { 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "result": [ 22:16:02 DEBUG { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "instance": "10.142.0.100:9100" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG [...] 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG "instance": "10.142.0.99:9100" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG "0.000333333333333267" 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG { 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "data": { 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "result": [ 22:16:02 DEBUG { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG [...] 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG "0" 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG { 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "data": { 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "result": [ 22:16:02 DEBUG { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "pod": "ds-cts-0" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG [...] 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG "pod": "ds-idrepo-0" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG "639.4666666666667" 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:02 INFO [http_cmd]: http status code OK 22:16:02 DEBUG --- status code --- 22:16:02 DEBUG http status code is 200 (expected 200) 22:16:02 DEBUG --- http response --- 22:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:16:02 DEBUG ----- output ----- 22:16:02 DEBUG { 22:16:02 DEBUG "status": "success", 22:16:02 DEBUG "data": { 22:16:02 DEBUG "resultType": "vector", 22:16:02 DEBUG "result": [ 22:16:02 DEBUG { 22:16:02 DEBUG "metric": { 22:16:02 DEBUG "pod": "ds-cts-0" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG [...] 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG 22:16:02 DEBUG "pod": "ds-cts-2" 22:16:02 DEBUG }, 22:16:02 DEBUG "value": [ 22:16:02 DEBUG 1686690962, 22:16:02 DEBUG "0" 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG ] 22:16:02 DEBUG } 22:16:02 DEBUG } 22:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690962" 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG { 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG "data": { 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG "result": [ 22:16:03 DEBUG { 22:16:03 DEBUG "metric": { 22:16:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG [...] 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 1686690962, 22:16:03 DEBUG "0" 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690963" 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690963" 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG { 22:16:03 DEBUG { 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG "data": { 22:16:03 DEBUG "data": { 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG "result": [ 22:16:03 DEBUG "result": [ 22:16:03 DEBUG { 22:16:03 DEBUG { 22:16:03 DEBUG "metric": { 22:16:03 DEBUG "metric": { 22:16:03 DEBUG "pod": "ds-cts-2" 22:16:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:16:03 DEBUG }, 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG [...] 22:16:03 DEBUG [...] 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:16:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:16:03 DEBUG }, 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG "0" 22:16:03 DEBUG "0" 22:16:03 DEBUG ] 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686690963" 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG { 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG "data": { 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG "result": [ 22:16:03 DEBUG { 22:16:03 DEBUG "metric": { 22:16:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG [...] 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG "13617565696" 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690963" 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG { 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG "data": { 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG "result": [ 22:16:03 DEBUG { 22:16:03 DEBUG "metric": { 22:16:03 DEBUG "pod": "ds-idrepo-2" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG [...] 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG "pod": "ds-idrepo-0" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG "639.4333333333333" 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690963" 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG { 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG "data": { 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG "result": [ 22:16:03 DEBUG { 22:16:03 DEBUG "metric": { 22:16:03 DEBUG "instance": "10.142.0.100:9100" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG [...] 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG "instance": "10.142.0.99:9100" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG "0.00050000000000144" 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690963" 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686690963" 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG { 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG "data": { 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG "result": [ 22:16:03 DEBUG { 22:16:03 DEBUG { 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG "metric": { 22:16:03 DEBUG "data": { 22:16:03 DEBUG "instance": "10.142.0.100:9100" 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG }, 22:16:03 DEBUG "result": [ 22:16:03 DEBUG "value": [ 22:16:03 DEBUG { 22:16:03 DEBUG 22:16:03 DEBUG "metric": { 22:16:03 DEBUG 22:16:03 DEBUG "instance": "10.142.0.100:9100" 22:16:03 DEBUG 22:16:03 DEBUG }, 22:16:03 DEBUG [...] 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG "instance": "10.142.0.99:9100" 22:16:03 DEBUG [...] 22:16:03 DEBUG }, 22:16:03 DEBUG 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG 22:16:03 DEBUG "20010532.2" 22:16:03 DEBUG "instance": "10.142.0.99:9100" 22:16:03 DEBUG ] 22:16:03 DEBUG }, 22:16:03 DEBUG } 22:16:03 DEBUG "value": [ 22:16:03 DEBUG ] 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG } 22:16:03 DEBUG "20008684.96666667" 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690963" 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG { 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG "data": { 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG "result": [ 22:16:03 DEBUG { 22:16:03 DEBUG "metric": { 22:16:03 DEBUG "pod": "ds-idrepo-2" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG [...] 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG "pod": "ds-idrepo-0" 22:16:03 DEBUG }, 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG "639.4333333333333" 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686690963" 22:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686690963" 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:16:03 INFO [http_cmd]: http status code OK 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG --- status code --- 22:16:03 DEBUG { 22:16:03 DEBUG http status code is 200 (expected 200) 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG --- http response --- 22:16:03 DEBUG "data": { 22:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG ----- output ----- 22:16:03 DEBUG "result": [ 22:16:03 DEBUG { 22:16:03 DEBUG { 22:16:03 DEBUG "status": "success", 22:16:03 DEBUG "metric": { 22:16:03 DEBUG "data": { 22:16:03 DEBUG "pod": "ds-cts-2" 22:16:03 DEBUG "resultType": "vector", 22:16:03 DEBUG }, 22:16:03 DEBUG "result": [ 22:16:03 DEBUG "value": [ 22:16:03 DEBUG { 22:16:03 DEBUG 22:16:03 DEBUG "metric": { 22:16:03 DEBUG 22:16:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:16:03 DEBUG 22:16:03 DEBUG }, 22:16:03 DEBUG [...] 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG 22:16:03 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:16:03 DEBUG [...] 22:16:03 DEBUG }, 22:16:03 DEBUG 22:16:03 DEBUG "value": [ 22:16:03 DEBUG 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG 22:16:03 DEBUG "0" 22:16:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:16:03 DEBUG ] 22:16:03 DEBUG }, 22:16:03 DEBUG } 22:16:03 DEBUG "value": [ 22:16:03 DEBUG ] 22:16:03 DEBUG 1686690963, 22:16:03 DEBUG } 22:16:03 DEBUG "0.2431508281991532" 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG ] 22:16:03 DEBUG } 22:16:03 DEBUG } 22:16:19 INFO 22:16:19 INFO [loop_until]: kubectl --namespace=xlou top pods 22:16:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:16:19 INFO [loop_until]: OK (rc = 0) 22:16:19 DEBUG --- stdout --- 22:16:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 10m 5754Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 10m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3408m 13800Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3132m 4908Mi idm-58dc667486-rgz9j 2567m 5094Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 207m 640Mi 22:16:19 DEBUG --- stderr --- 22:16:19 DEBUG 22:16:23 INFO 22:16:23 INFO [loop_until]: kubectl --namespace=xlou top node 22:16:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:16:23 INFO [loop_until]: OK (rc = 0) 22:16:23 DEBUG --- stdout --- 22:16:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3313m 20% 5851Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 74m 0% 6865Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 77m 0% 6826Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2708m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 449m 2% 3141Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 78m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 63m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1196Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13714Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3395m 21% 14505Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14594Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 270m 1% 1958Mi 3% 22:16:23 DEBUG --- stderr --- 22:16:23 DEBUG 22:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691022" 22:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691022" 22:17:02 INFO [http_cmd]: http status code OK 22:17:02 INFO [http_cmd]: http status code OK 22:17:02 DEBUG --- status code --- 22:17:02 DEBUG --- status code --- 22:17:02 DEBUG http status code is 200 (expected 200) 22:17:02 DEBUG http status code is 200 (expected 200) 22:17:02 DEBUG --- http response --- 22:17:02 DEBUG --- http response --- 22:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:02 DEBUG ----- output ----- 22:17:02 DEBUG ----- output ----- 22:17:02 DEBUG { 22:17:02 DEBUG { 22:17:02 DEBUG "status": "success", 22:17:02 DEBUG "status": "success", 22:17:02 DEBUG "data": { 22:17:02 DEBUG "data": { 22:17:02 DEBUG "resultType": "vector", 22:17:02 DEBUG "resultType": "vector", 22:17:02 DEBUG "result": [ 22:17:02 DEBUG "result": [ 22:17:02 DEBUG { 22:17:02 DEBUG { 22:17:02 DEBUG "metric": { 22:17:02 DEBUG "metric": { 22:17:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:17:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:17:02 DEBUG }, 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG [...] 22:17:02 DEBUG [...] 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:17:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:17:02 DEBUG }, 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 1686691022, 22:17:02 DEBUG 1686691022, 22:17:02 DEBUG "0" 22:17:02 DEBUG "0" 22:17:02 DEBUG ] 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:02 DEBUG ] 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691022" 22:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691022" 22:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691022" 22:17:02 INFO [http_cmd]: http status code OK 22:17:02 DEBUG --- status code --- 22:17:02 DEBUG http status code is 200 (expected 200) 22:17:02 DEBUG --- http response --- 22:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:02 DEBUG ----- output ----- 22:17:02 DEBUG { 22:17:02 DEBUG "status": "success", 22:17:02 DEBUG "data": { 22:17:02 DEBUG "resultType": "vector", 22:17:02 DEBUG "result": [ 22:17:02 DEBUG { 22:17:02 DEBUG "metric": { 22:17:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG [...] 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 1686691022, 22:17:02 DEBUG "0.2" 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:02 INFO [http_cmd]: http status code OK 22:17:02 DEBUG --- status code --- 22:17:02 DEBUG http status code is 200 (expected 200) 22:17:02 DEBUG --- http response --- 22:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:02 INFO [http_cmd]: http status code OK 22:17:02 DEBUG ----- output ----- 22:17:02 DEBUG --- status code --- 22:17:02 DEBUG { 22:17:02 DEBUG http status code is 200 (expected 200) 22:17:02 DEBUG "status": "success", 22:17:02 DEBUG --- http response --- 22:17:02 DEBUG "data": { 22:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:17:02 DEBUG "resultType": "vector", 22:17:02 DEBUG ----- output ----- 22:17:02 DEBUG "result": [ 22:17:02 DEBUG { 22:17:02 DEBUG { 22:17:02 DEBUG "status": "success", 22:17:02 DEBUG "metric": { 22:17:02 DEBUG "data": { 22:17:02 DEBUG "pod": "ds-idrepo-2" 22:17:02 DEBUG "resultType": "vector", 22:17:02 DEBUG }, 22:17:02 DEBUG "result": [ 22:17:02 DEBUG "value": [ 22:17:02 DEBUG { 22:17:02 DEBUG 22:17:02 DEBUG "metric": { 22:17:02 DEBUG 22:17:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:17:02 DEBUG 22:17:02 DEBUG }, 22:17:02 DEBUG [...] 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG "pod": "ds-idrepo-0" 22:17:02 DEBUG [...] 22:17:02 DEBUG }, 22:17:02 DEBUG 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 22:17:02 DEBUG 1686691022, 22:17:02 DEBUG 22:17:02 DEBUG "564.1333333333333" 22:17:02 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:17:02 DEBUG ] 22:17:02 DEBUG }, 22:17:02 DEBUG } 22:17:02 DEBUG "value": [ 22:17:02 DEBUG ] 22:17:02 DEBUG 1686691022, 22:17:02 DEBUG } 22:17:02 DEBUG "0.000032572385533582956" 22:17:02 DEBUG } 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691022" 22:17:02 INFO [http_cmd]: http status code OK 22:17:02 DEBUG --- status code --- 22:17:02 DEBUG http status code is 200 (expected 200) 22:17:02 DEBUG --- http response --- 22:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:17:02 DEBUG ----- output ----- 22:17:02 DEBUG { 22:17:02 DEBUG "status": "success", 22:17:02 DEBUG "data": { 22:17:02 DEBUG "resultType": "vector", 22:17:02 DEBUG "result": [ 22:17:02 DEBUG { 22:17:02 DEBUG "metric": { 22:17:02 DEBUG "pod": "ds-cts-0" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG [...] 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG "pod": "ds-idrepo-0" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 1686691022, 22:17:02 DEBUG "0" 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691022" 22:17:02 INFO [http_cmd]: http status code OK 22:17:02 DEBUG --- status code --- 22:17:02 DEBUG http status code is 200 (expected 200) 22:17:02 DEBUG --- http response --- 22:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:02 DEBUG ----- output ----- 22:17:02 DEBUG { 22:17:02 DEBUG "status": "success", 22:17:02 DEBUG "data": { 22:17:02 DEBUG "resultType": "vector", 22:17:02 DEBUG "result": [ 22:17:02 DEBUG { 22:17:02 DEBUG "metric": { 22:17:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG [...] 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 1686691022, 22:17:02 DEBUG "0" 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691022" 22:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691022" 22:17:02 INFO [http_cmd]: http status code OK 22:17:02 DEBUG --- status code --- 22:17:02 DEBUG http status code is 200 (expected 200) 22:17:02 DEBUG --- http response --- 22:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:02 DEBUG ----- output ----- 22:17:02 DEBUG { 22:17:02 DEBUG "status": "success", 22:17:02 DEBUG "data": { 22:17:02 DEBUG "resultType": "vector", 22:17:02 DEBUG "result": [ 22:17:02 DEBUG { 22:17:02 DEBUG "metric": { 22:17:02 DEBUG "instance": "10.142.0.100:9100" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG [...] 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG "instance": "10.142.0.99:9100" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 1686691022, 22:17:02 DEBUG "0" 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:02 INFO [http_cmd]: http status code OK 22:17:02 DEBUG --- status code --- 22:17:02 DEBUG http status code is 200 (expected 200) 22:17:02 DEBUG --- http response --- 22:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:17:02 DEBUG ----- output ----- 22:17:02 DEBUG { 22:17:02 DEBUG "status": "success", 22:17:02 DEBUG "data": { 22:17:02 DEBUG "resultType": "vector", 22:17:02 DEBUG "result": [ 22:17:02 DEBUG { 22:17:02 DEBUG "metric": { 22:17:02 DEBUG "pod": "ds-cts-0" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG [...] 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG 22:17:02 DEBUG "pod": "ds-idrepo-0" 22:17:02 DEBUG }, 22:17:02 DEBUG "value": [ 22:17:02 DEBUG 1686691022, 22:17:02 DEBUG "564.1333333333333" 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG ] 22:17:02 DEBUG } 22:17:02 DEBUG } 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691023" 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691023" 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "pod": "ds-cts-0" 22:17:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:17:03 DEBUG }, 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "pod": "ds-cts-2" 22:17:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:17:03 DEBUG }, 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "0" 22:17:03 DEBUG "0" 22:17:03 DEBUG ] 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691023" 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "0" 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691023" 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "0" 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691023" 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "13615960064" 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691023" 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "pod": "ds-idrepo-2" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "pod": "ds-idrepo-0" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "564.1" 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691023" 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691023" 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691023" 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "instance": "10.142.0.100:9100" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "instance": "10.142.0.99:9100" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "0.00036666666666557526" 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "instance": "10.142.0.100:9100" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "instance": "10.142.0.99:9100" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "18748923" 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "instance": "10.142.0.100:9100" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "instance": "10.142.0.99:9100" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "18752368.966666665" 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691023" 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "pod": "ds-idrepo-2" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "pod": "ds-idrepo-0" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "564.1" 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691023" 22:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691023" 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG { 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "data": { 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "result": [ 22:17:03 DEBUG { 22:17:03 DEBUG "metric": { 22:17:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 INFO [http_cmd]: http status code OK 22:17:03 DEBUG 22:17:03 DEBUG --- status code --- 22:17:03 DEBUG 22:17:03 DEBUG http status code is 200 (expected 200) 22:17:03 DEBUG 22:17:03 DEBUG --- http response --- 22:17:03 DEBUG [...] 22:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:03 DEBUG 22:17:03 DEBUG ----- output ----- 22:17:03 DEBUG 22:17:03 DEBUG { 22:17:03 DEBUG 22:17:03 DEBUG "status": "success", 22:17:03 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:17:03 DEBUG "data": { 22:17:03 DEBUG }, 22:17:03 DEBUG "resultType": "vector", 22:17:03 DEBUG "value": [ 22:17:03 DEBUG "result": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG { 22:17:03 DEBUG "0" 22:17:03 DEBUG "metric": { 22:17:03 DEBUG ] 22:17:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:17:03 DEBUG } 22:17:03 DEBUG }, 22:17:03 DEBUG ] 22:17:03 DEBUG "value": [ 22:17:03 DEBUG } 22:17:03 DEBUG 22:17:03 DEBUG } 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG [...] 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG 22:17:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:17:03 DEBUG }, 22:17:03 DEBUG "value": [ 22:17:03 DEBUG 1686691023, 22:17:03 DEBUG "0.2353342251312581" 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG ] 22:17:03 DEBUG } 22:17:03 DEBUG } 22:17:19 INFO 22:17:19 INFO [loop_until]: kubectl --namespace=xlou top pods 22:17:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:17:19 INFO [loop_until]: OK (rc = 0) 22:17:19 DEBUG --- stdout --- 22:17:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 10m 5754Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 9m 392Mi ds-idrepo-0 3427m 13830Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2982m 4909Mi idm-58dc667486-rgz9j 2686m 5095Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 195m 640Mi 22:17:19 DEBUG --- stderr --- 22:17:19 DEBUG 22:17:23 INFO 22:17:23 INFO [loop_until]: kubectl --namespace=xlou top node 22:17:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:17:23 INFO [loop_until]: OK (rc = 0) 22:17:23 DEBUG --- stdout --- 22:17:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3066m 19% 5852Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 78m 0% 6827Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2732m 17% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 424m 2% 3128Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 77m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13716Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3265m 20% 14505Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 68m 0% 14592Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 263m 1% 1955Mi 3% 22:17:23 DEBUG --- stderr --- 22:17:23 DEBUG 22:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691082" 22:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691082" 22:18:02 INFO [http_cmd]: http status code OK 22:18:02 DEBUG --- status code --- 22:18:02 DEBUG http status code is 200 (expected 200) 22:18:02 DEBUG --- http response --- 22:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:18:02 DEBUG ----- output ----- 22:18:02 DEBUG { 22:18:02 DEBUG "status": "success", 22:18:02 DEBUG "data": { 22:18:02 DEBUG "resultType": "vector", 22:18:02 DEBUG "result": [ 22:18:02 DEBUG { 22:18:02 DEBUG "metric": { 22:18:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG [...] 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 1686691082, 22:18:02 DEBUG "3616.727197682264" 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG } 22:18:02 INFO [http_cmd]: http status code OK 22:18:02 DEBUG --- status code --- 22:18:02 DEBUG http status code is 200 (expected 200) 22:18:02 DEBUG --- http response --- 22:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:02 DEBUG ----- output ----- 22:18:02 DEBUG { 22:18:02 DEBUG "status": "success", 22:18:02 DEBUG "data": { 22:18:02 DEBUG "resultType": "vector", 22:18:02 DEBUG "result": [ 22:18:02 DEBUG { 22:18:02 DEBUG "metric": { 22:18:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG [...] 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 1686691082, 22:18:02 DEBUG "0" 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG } 22:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691082" 22:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691082" 22:18:02 INFO [http_cmd]: http status code OK 22:18:02 DEBUG --- status code --- 22:18:02 DEBUG http status code is 200 (expected 200) 22:18:02 DEBUG --- http response --- 22:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:18:02 DEBUG ----- output ----- 22:18:02 DEBUG { 22:18:02 DEBUG "status": "success", 22:18:02 DEBUG "data": { 22:18:02 DEBUG "resultType": "vector", 22:18:02 DEBUG "result": [ 22:18:02 DEBUG { 22:18:02 DEBUG "metric": { 22:18:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG [...] 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 1686691082, 22:18:02 DEBUG "0.003884989483653447" 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG } 22:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691082" 22:18:02 INFO [http_cmd]: http status code OK 22:18:02 DEBUG --- status code --- 22:18:02 DEBUG http status code is 200 (expected 200) 22:18:02 DEBUG --- http response --- 22:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:02 DEBUG ----- output ----- 22:18:02 DEBUG { 22:18:02 DEBUG "status": "success", 22:18:02 DEBUG "data": { 22:18:02 DEBUG "resultType": "vector", 22:18:02 DEBUG "result": [ 22:18:02 DEBUG { 22:18:02 DEBUG "metric": { 22:18:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG [...] 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 1686691082, 22:18:02 DEBUG "0.2" 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG } 22:18:02 INFO [http_cmd]: http status code OK 22:18:02 DEBUG --- status code --- 22:18:02 DEBUG http status code is 200 (expected 200) 22:18:02 DEBUG --- http response --- 22:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:02 DEBUG ----- output ----- 22:18:02 DEBUG { 22:18:02 DEBUG "status": "success", 22:18:02 DEBUG "data": { 22:18:02 DEBUG "resultType": "vector", 22:18:02 DEBUG "result": [ 22:18:02 DEBUG { 22:18:02 DEBUG "metric": { 22:18:02 DEBUG "pod": "ds-idrepo-2" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG [...] 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG "pod": "ds-idrepo-0" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 1686691082, 22:18:02 DEBUG "657.7333333333333" 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG } 22:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691082" 22:18:02 INFO [http_cmd]: http status code OK 22:18:02 DEBUG --- status code --- 22:18:02 DEBUG http status code is 200 (expected 200) 22:18:02 DEBUG --- http response --- 22:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:18:02 DEBUG ----- output ----- 22:18:02 DEBUG { 22:18:02 DEBUG "status": "success", 22:18:02 DEBUG "data": { 22:18:02 DEBUG "resultType": "vector", 22:18:02 DEBUG "result": [ 22:18:02 DEBUG { 22:18:02 DEBUG "metric": { 22:18:02 DEBUG "pod": "ds-cts-0" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG [...] 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG 22:18:02 DEBUG "pod": "ds-idrepo-0" 22:18:02 DEBUG }, 22:18:02 DEBUG "value": [ 22:18:02 DEBUG 1686691082, 22:18:02 DEBUG "0" 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG ] 22:18:02 DEBUG } 22:18:02 DEBUG } 22:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691082" 22:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691082" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "pod": "ds-cts-0" 22:18:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:18:03 DEBUG }, 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "pod": "ds-idrepo-0" 22:18:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:18:03 DEBUG }, 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691082, 22:18:03 DEBUG 1686691082, 22:18:03 DEBUG "657.7666666666667" 22:18:03 DEBUG "0" 22:18:03 DEBUG ] 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691083" 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "0" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "instance": "10.142.0.100:9100" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "instance": "10.142.0.99:9100" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "0" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "pod": "ds-cts-0" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "pod": "ds-cts-2" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "0" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "0" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "0" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "13615984640" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "pod": "ds-idrepo-2" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "pod": "ds-idrepo-0" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "657.7333333333333" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "instance": "10.142.0.100:9100" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "instance": "10.142.0.99:9100" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "19785720.599999998" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691083" 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:03 DEBUG { 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG { 22:18:03 DEBUG "data": { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "result": [ 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG { 22:18:03 DEBUG "result": [ 22:18:03 DEBUG "metric": { 22:18:03 DEBUG { 22:18:03 DEBUG "instance": "10.142.0.100:9100" 22:18:03 DEBUG "metric": { 22:18:03 DEBUG }, 22:18:03 DEBUG "instance": "10.142.0.100:9100" 22:18:03 DEBUG "value": [ 22:18:03 DEBUG }, 22:18:03 DEBUG 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "instance": "10.142.0.99:9100" 22:18:03 DEBUG 22:18:03 DEBUG }, 22:18:03 DEBUG "instance": "10.142.0.99:9100" 22:18:03 DEBUG "value": [ 22:18:03 DEBUG }, 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG "0.00026666666666604516" 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG ] 22:18:03 DEBUG "19781126.5" 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "pod": "ds-idrepo-2" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "pod": "ds-idrepo-0" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "657.8" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "0" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691083" 22:18:03 INFO [http_cmd]: http status code OK 22:18:03 DEBUG --- status code --- 22:18:03 DEBUG http status code is 200 (expected 200) 22:18:03 DEBUG --- http response --- 22:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:03 DEBUG ----- output ----- 22:18:03 DEBUG { 22:18:03 DEBUG "status": "success", 22:18:03 DEBUG "data": { 22:18:03 DEBUG "resultType": "vector", 22:18:03 DEBUG "result": [ 22:18:03 DEBUG { 22:18:03 DEBUG "metric": { 22:18:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG [...] 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG 22:18:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:18:03 DEBUG }, 22:18:03 DEBUG "value": [ 22:18:03 DEBUG 1686691083, 22:18:03 DEBUG "0.24663692976443902" 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG ] 22:18:03 DEBUG } 22:18:03 DEBUG } 22:18:19 INFO 22:18:19 INFO [loop_until]: kubectl --namespace=xlou top pods 22:18:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:18:19 INFO [loop_until]: OK (rc = 0) 22:18:20 DEBUG --- stdout --- 22:18:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5754Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3193m 13822Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3124m 4911Mi idm-58dc667486-rgz9j 2790m 5095Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 206m 640Mi 22:18:20 DEBUG --- stderr --- 22:18:20 DEBUG 22:18:23 INFO 22:18:23 INFO [loop_until]: kubectl --namespace=xlou top node 22:18:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:18:23 INFO [loop_until]: OK (rc = 0) 22:18:23 DEBUG --- stdout --- 22:18:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3163m 19% 5868Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2771m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 442m 2% 3125Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 78m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1196Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13718Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3381m 21% 14503Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14599Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 270m 1% 1960Mi 3% 22:18:23 DEBUG --- stderr --- 22:18:23 DEBUG 22:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691142" 22:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691142" 22:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691142" 22:19:02 INFO [http_cmd]: http status code OK 22:19:02 DEBUG --- status code --- 22:19:02 DEBUG http status code is 200 (expected 200) 22:19:02 INFO [http_cmd]: http status code OK 22:19:02 DEBUG --- http response --- 22:19:02 DEBUG --- status code --- 22:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:02 DEBUG http status code is 200 (expected 200) 22:19:02 DEBUG ----- output ----- 22:19:02 DEBUG --- http response --- 22:19:02 DEBUG { 22:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:19:02 DEBUG "status": "success", 22:19:02 DEBUG ----- output ----- 22:19:02 DEBUG "data": { 22:19:02 DEBUG { 22:19:02 DEBUG "resultType": "vector", 22:19:02 DEBUG "status": "success", 22:19:02 DEBUG "result": [ 22:19:02 DEBUG "data": { 22:19:02 DEBUG { 22:19:02 DEBUG "resultType": "vector", 22:19:02 DEBUG "metric": { 22:19:02 DEBUG "result": [ 22:19:02 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:19:02 DEBUG { 22:19:02 DEBUG }, 22:19:02 DEBUG "metric": { 22:19:02 DEBUG "value": [ 22:19:02 DEBUG "pod": "ds-cts-2" 22:19:02 DEBUG 22:19:02 DEBUG }, 22:19:02 DEBUG 22:19:02 DEBUG "value": [ 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG [...] 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG [...] 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:19:02 DEBUG 22:19:02 DEBUG }, 22:19:02 DEBUG 22:19:02 DEBUG "value": [ 22:19:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:19:02 DEBUG 1686691142, 22:19:02 DEBUG }, 22:19:02 DEBUG "0" 22:19:02 DEBUG "value": [ 22:19:02 DEBUG ] 22:19:02 DEBUG 1686691142, 22:19:02 DEBUG } 22:19:02 DEBUG "0" 22:19:02 DEBUG ] 22:19:02 DEBUG ] 22:19:02 DEBUG } 22:19:02 DEBUG } 22:19:02 DEBUG } 22:19:02 DEBUG ] 22:19:02 DEBUG } 22:19:02 DEBUG } 22:19:02 INFO [http_cmd]: http status code OK 22:19:02 DEBUG --- status code --- 22:19:02 DEBUG http status code is 200 (expected 200) 22:19:02 DEBUG --- http response --- 22:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:19:02 DEBUG ----- output ----- 22:19:02 DEBUG { 22:19:02 DEBUG "status": "success", 22:19:02 DEBUG "data": { 22:19:02 DEBUG "resultType": "vector", 22:19:02 DEBUG "result": [ 22:19:02 DEBUG { 22:19:02 DEBUG "metric": { 22:19:02 DEBUG "pod": "ds-cts-2" 22:19:02 DEBUG }, 22:19:02 DEBUG "value": [ 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG [...] 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:19:02 DEBUG }, 22:19:02 DEBUG "value": [ 22:19:02 DEBUG 1686691142, 22:19:02 DEBUG "0.00003245099216274457" 22:19:02 DEBUG ] 22:19:02 DEBUG } 22:19:02 DEBUG ] 22:19:02 DEBUG } 22:19:02 DEBUG } 22:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691142" 22:19:02 INFO [http_cmd]: http status code OK 22:19:02 DEBUG --- status code --- 22:19:02 DEBUG http status code is 200 (expected 200) 22:19:02 DEBUG --- http response --- 22:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:19:02 DEBUG ----- output ----- 22:19:02 DEBUG { 22:19:02 DEBUG "status": "success", 22:19:02 DEBUG "data": { 22:19:02 DEBUG "resultType": "vector", 22:19:02 DEBUG "result": [ 22:19:02 DEBUG { 22:19:02 DEBUG "metric": { 22:19:02 DEBUG "pod": "ds-cts-0" 22:19:02 DEBUG }, 22:19:02 DEBUG "value": [ 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG [...] 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG 22:19:02 DEBUG "pod": "ds-idrepo-0" 22:19:02 DEBUG }, 22:19:02 DEBUG "value": [ 22:19:02 DEBUG 1686691142, 22:19:02 DEBUG "0" 22:19:02 DEBUG ] 22:19:02 DEBUG } 22:19:02 DEBUG ] 22:19:02 DEBUG } 22:19:02 DEBUG } 22:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691142" 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "pod": "ds-idrepo-2" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "pod": "ds-idrepo-0" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691142, 22:19:03 DEBUG "538.0333333333333" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "0.2" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:19:03 DEBUG "pod": "ds-cts-0" 22:19:03 DEBUG }, 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:19:03 DEBUG "pod": "ds-idrepo-0" 22:19:03 DEBUG }, 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "0" 22:19:03 DEBUG "537.9666666666667" 22:19:03 DEBUG ] 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "pod": "ds-cts-0" 22:19:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:19:03 DEBUG }, 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "pod": "ds-cts-2" 22:19:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:19:03 DEBUG }, 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "0" 22:19:03 DEBUG "0" 22:19:03 DEBUG ] 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "instance": "10.142.0.100:9100" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "instance": "10.142.0.99:9100" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "0" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "pod": "ds-cts-2" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "0" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "0" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "13616738304" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "pod": "ds-idrepo-2" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "pod": "ds-idrepo-0" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "538.0666666666666" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "instance": "10.142.0.100:9100" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "instance": "10.142.0.99:9100" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "20561919.933333334" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691143" 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "instance": "10.142.0.100:9100" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "instance": "10.142.0.99:9100" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "0.00030000000000048505" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "instance": "10.142.0.100:9100" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "instance": "10.142.0.99:9100" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "20557915.9" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "pod": "ds-idrepo-2" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "pod": "ds-idrepo-0" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "538.0666666666666" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691143" 22:19:03 INFO [http_cmd]: http status code OK 22:19:03 DEBUG --- status code --- 22:19:03 DEBUG http status code is 200 (expected 200) 22:19:03 DEBUG --- http response --- 22:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:03 DEBUG ----- output ----- 22:19:03 DEBUG { 22:19:03 DEBUG "status": "success", 22:19:03 DEBUG "data": { 22:19:03 DEBUG "resultType": "vector", 22:19:03 DEBUG "result": [ 22:19:03 DEBUG { 22:19:03 DEBUG "metric": { 22:19:03 DEBUG "pod": "ds-cts-2" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG [...] 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG 22:19:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:19:03 DEBUG }, 22:19:03 DEBUG "value": [ 22:19:03 DEBUG 1686691143, 22:19:03 DEBUG "0" 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG ] 22:19:03 DEBUG } 22:19:03 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691144" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected 200) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1686691144, 22:19:04 DEBUG "0.23426917322106522" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:20 INFO 22:19:20 INFO [loop_until]: kubectl --namespace=xlou top pods 22:19:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:19:20 INFO [loop_until]: OK (rc = 0) 22:19:20 DEBUG --- stdout --- 22:19:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5755Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 392Mi ds-idrepo-0 3415m 13804Mi ds-idrepo-1 14m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3133m 4913Mi idm-58dc667486-rgz9j 2782m 5095Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 202m 641Mi 22:19:20 DEBUG --- stderr --- 22:19:20 DEBUG 22:19:23 INFO 22:19:23 INFO [loop_until]: kubectl --namespace=xlou top node 22:19:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:19:23 INFO [loop_until]: OK (rc = 0) 22:19:23 DEBUG --- stdout --- 22:19:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3344m 21% 5858Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 74m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6834Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2817m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 428m 2% 3131Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1482Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13717Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3487m 21% 14509Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 69m 0% 14601Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 267m 1% 1959Mi 3% 22:19:23 DEBUG --- stderr --- 22:19:23 DEBUG 22:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691202" 22:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691202" 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:20:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:20:03 DEBUG }, 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:20:03 DEBUG 22:20:03 DEBUG }, 22:20:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:20:03 DEBUG "value": [ 22:20:03 DEBUG }, 22:20:03 DEBUG 1686691202, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG "0" 22:20:03 DEBUG 1686691202, 22:20:03 DEBUG ] 22:20:03 DEBUG "0.005626598909170315" 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "152231925.36715978" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "ds-cts-0" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "ds-idrepo-0" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "0" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "ds-idrepo-2" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "ds-idrepo-0" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "632.8666666666667" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "0.2" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "0" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "ds-cts-0" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "ds-idrepo-0" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "632.8666666666667" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:20:03 DEBUG "pod": "ds-cts-0" 22:20:03 DEBUG }, 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:20:03 DEBUG "pod": "ds-cts-2" 22:20:03 DEBUG }, 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "0" 22:20:03 DEBUG "0" 22:20:03 DEBUG ] 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "instance": "10.142.0.100:9100" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "instance": "10.142.0.99:9100" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "0" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "0" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG { 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG "metric": { 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG }, 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:03 DEBUG "value": [ 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG 22:20:03 DEBUG { 22:20:03 DEBUG 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG 22:20:03 DEBUG "data": { 22:20:03 DEBUG [...] 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG 22:20:03 DEBUG "result": [ 22:20:03 DEBUG 22:20:03 DEBUG { 22:20:03 DEBUG 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:20:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:20:03 DEBUG }, 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG 22:20:03 DEBUG "0" 22:20:03 DEBUG 22:20:03 DEBUG ] 22:20:03 DEBUG 22:20:03 DEBUG } 22:20:03 DEBUG [...] 22:20:03 DEBUG ] 22:20:03 DEBUG 22:20:03 DEBUG } 22:20:03 DEBUG 22:20:03 DEBUG } 22:20:03 DEBUG 22:20:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "13617188864" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "ds-idrepo-2" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "ds-idrepo-0" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "632.8666666666667" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "instance": "10.142.0.100:9100" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "instance": "10.142.0.99:9100" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "20383108.133333333" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691203" 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "instance": "10.142.0.100:9100" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "instance": "10.142.0.99:9100" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "0.00036666666666747006" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "instance": "10.142.0.100:9100" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "instance": "10.142.0.99:9100" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "20379287.366666667" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691203" 22:20:03 INFO [http_cmd]: http status code OK 22:20:03 DEBUG --- status code --- 22:20:03 DEBUG http status code is 200 (expected 200) 22:20:03 DEBUG --- http response --- 22:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:03 DEBUG ----- output ----- 22:20:03 DEBUG { 22:20:03 DEBUG "status": "success", 22:20:03 DEBUG "data": { 22:20:03 DEBUG "resultType": "vector", 22:20:03 DEBUG "result": [ 22:20:03 DEBUG { 22:20:03 DEBUG "metric": { 22:20:03 DEBUG "pod": "ds-idrepo-2" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG [...] 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG 22:20:03 DEBUG "pod": "ds-idrepo-0" 22:20:03 DEBUG }, 22:20:03 DEBUG "value": [ 22:20:03 DEBUG 1686691203, 22:20:03 DEBUG "632.8333333333334" 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG ] 22:20:03 DEBUG } 22:20:03 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691204" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected 200) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1686691204, 22:20:04 DEBUG "0" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691204" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected 200) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1686691204, 22:20:04 DEBUG "0.23451110143267995" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:20 INFO 22:20:20 INFO [loop_until]: kubectl --namespace=xlou top pods 22:20:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:20:20 INFO [loop_until]: OK (rc = 0) 22:20:20 DEBUG --- stdout --- 22:20:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5757Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 8m 425Mi ds-cts-2 8m 392Mi ds-idrepo-0 3346m 13807Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3283m 4915Mi idm-58dc667486-rgz9j 2691m 5095Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 214m 641Mi 22:20:20 DEBUG --- stderr --- 22:20:20 DEBUG 22:20:23 INFO 22:20:23 INFO [loop_until]: kubectl --namespace=xlou top node 22:20:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:20:23 INFO [loop_until]: OK (rc = 0) 22:20:23 DEBUG --- stdout --- 22:20:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3313m 20% 5856Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 65m 0% 6860Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 76m 0% 6832Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2728m 17% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6952Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 453m 2% 3126Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1483Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1200Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13716Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3476m 21% 14505Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 60m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14596Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 274m 1% 1958Mi 3% 22:20:23 DEBUG --- stderr --- 22:20:23 DEBUG 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:21:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 DEBUG }, 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:21:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:21:03 DEBUG }, 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "0" 22:21:03 DEBUG "0" 22:21:03 DEBUG ] 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "ds-cts-0" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "ds-idrepo-0" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "0" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "0" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "ds-idrepo-2" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "ds-idrepo-0" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "664.5666666666666" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "0.2" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "ds-cts-0" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "ds-idrepo-0" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "664.6" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG { 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG "data": { 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:21:03 DEBUG "result": [ 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:21:03 DEBUG "data": { 22:21:03 DEBUG }, 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "value": [ 22:21:03 DEBUG "result": [ 22:21:03 DEBUG 22:21:03 DEBUG { 22:21:03 DEBUG 22:21:03 DEBUG "metric": { 22:21:03 DEBUG 22:21:03 DEBUG "pod": "ds-cts-0" 22:21:03 DEBUG [...] 22:21:03 DEBUG }, 22:21:03 DEBUG 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:21:03 DEBUG 22:21:03 DEBUG }, 22:21:03 DEBUG [...] 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG 22:21:03 DEBUG "0" 22:21:03 DEBUG 22:21:03 DEBUG ] 22:21:03 DEBUG "pod": "ds-cts-2" 22:21:03 DEBUG } 22:21:03 DEBUG }, 22:21:03 DEBUG ] 22:21:03 DEBUG "value": [ 22:21:03 DEBUG } 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG } 22:21:03 DEBUG "0" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "0" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG } 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG } 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "instance": "10.142.0.100:9100" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "instance": "10.142.0.99:9100" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "0" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "ds-idrepo-2" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "0" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "0" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "13618802688" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "ds-idrepo-2" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "ds-idrepo-0" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "664.5333333333333" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "instance": "10.142.0.100:9100" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "instance": "10.142.0.99:9100" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "18853147.2" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691263" 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "instance": "10.142.0.100:9100" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "instance": "10.142.0.99:9100" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "18850890.266666666" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "instance": "10.142.0.100:9100" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "instance": "10.142.0.99:9100" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "0.0005333333333339852" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691263" 22:21:03 INFO [http_cmd]: http status code OK 22:21:03 DEBUG --- status code --- 22:21:03 DEBUG http status code is 200 (expected 200) 22:21:03 DEBUG --- http response --- 22:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:03 DEBUG ----- output ----- 22:21:03 DEBUG { 22:21:03 DEBUG "status": "success", 22:21:03 DEBUG "data": { 22:21:03 DEBUG "resultType": "vector", 22:21:03 DEBUG "result": [ 22:21:03 DEBUG { 22:21:03 DEBUG "metric": { 22:21:03 DEBUG "pod": "ds-idrepo-2" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG [...] 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG 22:21:03 DEBUG "pod": "ds-idrepo-0" 22:21:03 DEBUG }, 22:21:03 DEBUG "value": [ 22:21:03 DEBUG 1686691263, 22:21:03 DEBUG "664.5333333333333" 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG ] 22:21:03 DEBUG } 22:21:03 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691264" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected 200) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "ds-idrepo-2" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1686691264, 22:21:04 DEBUG "0.4764541779426014" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691264" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected 200) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1686691264, 22:21:04 DEBUG "0.23813156653174433" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:20 INFO 22:21:20 INFO [loop_until]: kubectl --namespace=xlou top pods 22:21:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:21:20 INFO [loop_until]: OK (rc = 0) 22:21:20 DEBUG --- stdout --- 22:21:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5757Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 424Mi ds-cts-2 8m 392Mi ds-idrepo-0 3292m 13804Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 15m 12909Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2920m 4917Mi idm-58dc667486-rgz9j 2755m 5095Mi lodemon-6455d5ff7c-zzndq 1m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 198m 641Mi 22:21:20 DEBUG --- stderr --- 22:21:20 DEBUG 22:21:23 INFO 22:21:23 INFO [loop_until]: kubectl --namespace=xlou top node 22:21:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:21:23 INFO [loop_until]: OK (rc = 0) 22:21:23 DEBUG --- stdout --- 22:21:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3034m 19% 5857Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2727m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6946Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 429m 2% 3132Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 77m 0% 1484Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 63m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 58m 0% 1192Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13718Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3446m 21% 14523Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14598Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 258m 1% 1959Mi 3% 22:21:23 DEBUG --- stderr --- 22:21:23 DEBUG 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "ds-cts-2" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "ds-cts-2" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0.00003189248434238314" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "ds-cts-0" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "ds-idrepo-0" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "ds-idrepo-2" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "ds-idrepo-0" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "628.0666666666666" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0.2" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0.03333333333333333" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "ds-cts-0" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "ds-cts-2" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "ds-cts-0" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "ds-idrepo-0" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "628.1" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0.03333333333333333" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "instance": "10.142.0.100:9100" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "instance": "10.142.0.99:9100" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "ds-cts-2" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "13616820224" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "ds-idrepo-2" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "ds-idrepo-0" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "628.0666666666666" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "instance": "10.142.0.100:9100" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "instance": "10.142.0.99:9100" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "19720213.566666666" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "instance": "10.142.0.100:9100" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "instance": "10.142.0.99:9100" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "0.00026666666666793997" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "instance": "10.142.0.100:9100" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "instance": "10.142.0.99:9100" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "19715974.4" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691323" 22:22:03 INFO [http_cmd]: http status code OK 22:22:03 DEBUG --- status code --- 22:22:03 DEBUG http status code is 200 (expected 200) 22:22:03 DEBUG --- http response --- 22:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:03 DEBUG ----- output ----- 22:22:03 DEBUG { 22:22:03 DEBUG "status": "success", 22:22:03 DEBUG "data": { 22:22:03 DEBUG "resultType": "vector", 22:22:03 DEBUG "result": [ 22:22:03 DEBUG { 22:22:03 DEBUG "metric": { 22:22:03 DEBUG "pod": "ds-idrepo-2" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG [...] 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG 22:22:03 DEBUG "pod": "ds-idrepo-0" 22:22:03 DEBUG }, 22:22:03 DEBUG "value": [ 22:22:03 DEBUG 1686691323, 22:22:03 DEBUG "628.0333333333333" 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG ] 22:22:03 DEBUG } 22:22:03 DEBUG } 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691324" 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected 200) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "pod": "ds-cts-2" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1686691324, 22:22:04 DEBUG "0" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691324" 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected 200) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1686691324, 22:22:04 DEBUG "0.23966665121000594" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:20 INFO 22:22:20 INFO [loop_until]: kubectl --namespace=xlou top pods 22:22:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:22:20 INFO [loop_until]: OK (rc = 0) 22:22:20 DEBUG --- stdout --- 22:22:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5757Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 392Mi ds-idrepo-0 3403m 13803Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2920m 4919Mi idm-58dc667486-rgz9j 2947m 5096Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 207m 641Mi 22:22:20 DEBUG --- stderr --- 22:22:20 DEBUG 22:22:23 INFO 22:22:23 INFO [loop_until]: kubectl --namespace=xlou top node 22:22:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:22:24 INFO [loop_until]: OK (rc = 0) 22:22:24 DEBUG --- stdout --- 22:22:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2929m 18% 5860Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 63m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 78m 0% 6828Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 3072m 19% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 456m 2% 3131Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 63m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 65m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 66m 0% 13714Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3432m 21% 14503Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14599Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 272m 1% 1958Mi 3% 22:22:24 DEBUG --- stderr --- 22:22:24 DEBUG 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "ds-cts-0" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "ds-idrepo-0" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "ds-idrepo-2" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "ds-idrepo-0" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "652.7" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "ds-cts-0" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "ds-idrepo-0" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "652.7" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0.2" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "ds-cts-0" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "ds-cts-2" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691383" 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "instance": "10.142.0.100:9100" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "instance": "10.142.0.99:9100" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0.0006666666666667116" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "13613793280" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "ds-idrepo-2" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "ds-idrepo-0" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "652.6666666666666" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "instance": "10.142.0.100:9100" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "instance": "10.142.0.99:9100" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "21987007.633333333" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "instance": "10.142.0.100:9100" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "instance": "10.142.0.99:9100" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "0.0003333333333330302" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "instance": "10.142.0.100:9100" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "instance": "10.142.0.99:9100" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "21982793.366666667" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691383" 22:23:03 INFO [http_cmd]: http status code OK 22:23:03 DEBUG --- status code --- 22:23:03 DEBUG http status code is 200 (expected 200) 22:23:03 DEBUG --- http response --- 22:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:03 DEBUG ----- output ----- 22:23:03 DEBUG { 22:23:03 DEBUG "status": "success", 22:23:03 DEBUG "data": { 22:23:03 DEBUG "resultType": "vector", 22:23:03 DEBUG "result": [ 22:23:03 DEBUG { 22:23:03 DEBUG "metric": { 22:23:03 DEBUG "pod": "ds-idrepo-2" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG [...] 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG 22:23:03 DEBUG "pod": "ds-idrepo-0" 22:23:03 DEBUG }, 22:23:03 DEBUG "value": [ 22:23:03 DEBUG 1686691383, 22:23:03 DEBUG "652.6666666666666" 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG ] 22:23:03 DEBUG } 22:23:03 DEBUG } 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691384" 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG http status code is 200 (expected 200) 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:23:04 DEBUG ----- output ----- 22:23:04 DEBUG { 22:23:04 DEBUG "status": "success", 22:23:04 DEBUG "data": { 22:23:04 DEBUG "resultType": "vector", 22:23:04 DEBUG "result": [ 22:23:04 DEBUG { 22:23:04 DEBUG "metric": { 22:23:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG [...] 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG "pod": "lodemon-6455d5ff7c-zzndq" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 1686691384, 22:23:04 DEBUG "0" 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG } 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691384" 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG http status code is 200 (expected 200) 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:04 DEBUG ----- output ----- 22:23:04 DEBUG { 22:23:04 DEBUG "status": "success", 22:23:04 DEBUG "data": { 22:23:04 DEBUG "resultType": "vector", 22:23:04 DEBUG "result": [ 22:23:04 DEBUG { 22:23:04 DEBUG "metric": { 22:23:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG [...] 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 1686691384, 22:23:04 DEBUG "0.2349427540971884" 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG } 22:23:20 INFO 22:23:20 INFO [loop_until]: kubectl --namespace=xlou top pods 22:23:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:23:20 INFO [loop_until]: OK (rc = 0) 22:23:20 DEBUG --- stdout --- 22:23:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5757Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 392Mi ds-idrepo-0 3647m 13797Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3414m 4921Mi idm-58dc667486-rgz9j 2446m 5096Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 218m 640Mi 22:23:20 DEBUG --- stderr --- 22:23:20 DEBUG 22:23:24 INFO 22:23:24 INFO [loop_until]: kubectl --namespace=xlou top node 22:23:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:23:24 INFO [loop_until]: OK (rc = 0) 22:23:24 DEBUG --- stdout --- 22:23:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3583m 22% 5864Mi 9% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2557m 16% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 428m 2% 3131Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 83m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 68m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13720Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3698m 23% 14527Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 69m 0% 14599Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 279m 1% 1953Mi 3% 22:23:24 DEBUG --- stderr --- 22:23:24 DEBUG 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG { 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG "data": { 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "data": { 22:24:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG }, 22:24:03 DEBUG "result": [ 22:24:03 DEBUG "value": [ 22:24:03 DEBUG { 22:24:03 DEBUG 22:24:03 DEBUG "metric": { 22:24:03 DEBUG 22:24:03 DEBUG "pod": "ds-cts-0" 22:24:03 DEBUG 22:24:03 DEBUG }, 22:24:03 DEBUG [...] 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:24:03 DEBUG [...] 22:24:03 DEBUG }, 22:24:03 DEBUG 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG 22:24:03 DEBUG "0" 22:24:03 DEBUG "pod": "ds-idrepo-0" 22:24:03 DEBUG ] 22:24:03 DEBUG }, 22:24:03 DEBUG } 22:24:03 DEBUG "value": [ 22:24:03 DEBUG ] 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG } 22:24:03 DEBUG "0" 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "ds-cts-2" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "0.00002835866242816934" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "ds-cts-2" 22:24:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:24:03 DEBUG }, 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:24:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:24:03 DEBUG }, 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "0" 22:24:03 DEBUG "0" 22:24:03 DEBUG ] 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "ds-idrepo-2" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "ds-idrepo-0" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "641.9" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "ds-cts-0" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "ds-cts-2" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "0" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "ds-cts-0" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "ds-idrepo-0" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "641.8666666666667" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:24:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:24:03 DEBUG }, 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:24:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:24:03 DEBUG }, 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "0.2" 22:24:03 DEBUG "0" 22:24:03 DEBUG ] 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "instance": "10.142.0.100:9100" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "instance": "10.142.0.99:9100" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "0" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "ds-cts-2" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "0" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "0" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "13615337472" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "ds-idrepo-2" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "ds-idrepo-0" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "641.8666666666667" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691443" 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "instance": "10.142.0.100:9100" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "instance": "10.142.0.99:9100" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "0.00040000000000001514" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "instance": "10.142.0.100:9100" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "instance": "10.142.0.99:9100" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "20594033.666666664" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "instance": "10.142.0.100:9100" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "instance": "10.142.0.99:9100" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "20589757.6" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691443" 22:24:03 INFO [http_cmd]: http status code OK 22:24:03 DEBUG --- status code --- 22:24:03 DEBUG http status code is 200 (expected 200) 22:24:03 DEBUG --- http response --- 22:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:03 DEBUG ----- output ----- 22:24:03 DEBUG { 22:24:03 DEBUG "status": "success", 22:24:03 DEBUG "data": { 22:24:03 DEBUG "resultType": "vector", 22:24:03 DEBUG "result": [ 22:24:03 DEBUG { 22:24:03 DEBUG "metric": { 22:24:03 DEBUG "pod": "ds-idrepo-2" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG [...] 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG 22:24:03 DEBUG "pod": "ds-idrepo-0" 22:24:03 DEBUG }, 22:24:03 DEBUG "value": [ 22:24:03 DEBUG 1686691443, 22:24:03 DEBUG "641.8666666666667" 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG ] 22:24:03 DEBUG } 22:24:03 DEBUG } 22:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691444" 22:24:04 INFO [http_cmd]: http status code OK 22:24:04 DEBUG --- status code --- 22:24:04 DEBUG http status code is 200 (expected 200) 22:24:04 DEBUG --- http response --- 22:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:04 DEBUG ----- output ----- 22:24:04 DEBUG { 22:24:04 DEBUG "status": "success", 22:24:04 DEBUG "data": { 22:24:04 DEBUG "resultType": "vector", 22:24:04 DEBUG "result": [ 22:24:04 DEBUG { 22:24:04 DEBUG "metric": { 22:24:04 DEBUG "pod": "ds-cts-2" 22:24:04 DEBUG }, 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG [...] 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:24:04 DEBUG }, 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 1686691444, 22:24:04 DEBUG "0" 22:24:04 DEBUG ] 22:24:04 DEBUG } 22:24:04 DEBUG ] 22:24:04 DEBUG } 22:24:04 DEBUG } 22:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691444" 22:24:04 INFO [http_cmd]: http status code OK 22:24:04 DEBUG --- status code --- 22:24:04 DEBUG http status code is 200 (expected 200) 22:24:04 DEBUG --- http response --- 22:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:04 DEBUG ----- output ----- 22:24:04 DEBUG { 22:24:04 DEBUG "status": "success", 22:24:04 DEBUG "data": { 22:24:04 DEBUG "resultType": "vector", 22:24:04 DEBUG "result": [ 22:24:04 DEBUG { 22:24:04 DEBUG "metric": { 22:24:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:24:04 DEBUG }, 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG [...] 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:24:04 DEBUG }, 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 1686691444, 22:24:04 DEBUG "0.24626060340853373" 22:24:04 DEBUG ] 22:24:04 DEBUG } 22:24:04 DEBUG ] 22:24:04 DEBUG } 22:24:04 DEBUG } 22:24:20 INFO 22:24:20 INFO [loop_until]: kubectl --namespace=xlou top pods 22:24:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:24:20 INFO [loop_until]: OK (rc = 0) 22:24:20 DEBUG --- stdout --- 22:24:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5757Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3370m 13822Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3171m 4923Mi idm-58dc667486-rgz9j 2896m 5096Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 207m 641Mi 22:24:20 DEBUG --- stderr --- 22:24:20 DEBUG 22:24:24 INFO 22:24:24 INFO [loop_until]: kubectl --namespace=xlou top node 22:24:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:24:24 INFO [loop_until]: OK (rc = 0) 22:24:24 DEBUG --- stdout --- 22:24:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3423m 21% 5865Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6854Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2506m 15% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 438m 2% 3126Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 86m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 65m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13719Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3439m 21% 14523Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1171Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14597Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 267m 1% 1956Mi 3% 22:24:24 DEBUG --- stderr --- 22:24:24 DEBUG 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG { 22:25:03 DEBUG { 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "data": { 22:25:03 DEBUG "data": { 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG "result": [ 22:25:03 DEBUG "result": [ 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG { 22:25:03 DEBUG { 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:25:03 DEBUG "pod": "ds-cts-0" 22:25:03 DEBUG "pod": "ds-cts-2" 22:25:03 DEBUG "pod": "ds-cts-2" 22:25:03 DEBUG }, 22:25:03 DEBUG }, 22:25:03 DEBUG }, 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG "value": [ 22:25:03 DEBUG "value": [ 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG [...] 22:25:03 DEBUG [...] 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:25:03 DEBUG "pod": "ds-idrepo-0" 22:25:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:25:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:25:03 DEBUG }, 22:25:03 DEBUG }, 22:25:03 DEBUG }, 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG "value": [ 22:25:03 DEBUG "value": [ 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "0" 22:25:03 DEBUG "0" 22:25:03 DEBUG "0" 22:25:03 DEBUG "0.0000277096464373661" 22:25:03 DEBUG ] 22:25:03 DEBUG ] 22:25:03 DEBUG ] 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG ] 22:25:03 DEBUG ] 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "0" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "pod": "ds-idrepo-2" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "pod": "ds-idrepo-0" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "661.0666666666666" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "pod": "ds-cts-0" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "pod": "ds-cts-2" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "0" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691503" 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG }, 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG "value": [ 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:25:03 DEBUG "0" 22:25:03 DEBUG ----- output ----- 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG ] 22:25:03 DEBUG { 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG } 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG ] 22:25:03 DEBUG "data": { 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG } 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:03 DEBUG } 22:25:03 DEBUG "result": [ 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "pod": "ds-cts-0" 22:25:03 DEBUG "data": { 22:25:03 DEBUG }, 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "value": [ 22:25:03 DEBUG "result": [ 22:25:03 DEBUG 22:25:03 DEBUG { 22:25:03 DEBUG 22:25:03 DEBUG "metric": { 22:25:03 DEBUG 22:25:03 DEBUG "instance": "10.142.0.100:9100" 22:25:03 DEBUG [...] 22:25:03 DEBUG }, 22:25:03 DEBUG 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "pod": "ds-idrepo-0" 22:25:03 DEBUG 22:25:03 DEBUG }, 22:25:03 DEBUG [...] 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG 22:25:03 DEBUG "661.0666666666666" 22:25:03 DEBUG 22:25:03 DEBUG ] 22:25:03 DEBUG "instance": "10.142.0.99:9100" 22:25:03 DEBUG } 22:25:03 DEBUG }, 22:25:03 DEBUG ] 22:25:03 DEBUG "value": [ 22:25:03 DEBUG } 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG } 22:25:03 DEBUG "0.000333333333333267" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "0.2" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "pod": "ds-cts-2" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "0" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "13618872320" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "0" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "pod": "ds-idrepo-2" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "pod": "ds-idrepo-0" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "661.1" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691503" 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "instance": "10.142.0.100:9100" 22:25:03 DEBUG "instance": "10.142.0.100:9100" 22:25:03 DEBUG }, 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "instance": "10.142.0.99:9100" 22:25:03 DEBUG "instance": "10.142.0.99:9100" 22:25:03 DEBUG }, 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "0.0004999999999995453" 22:25:03 DEBUG "21287837.633333333" 22:25:03 DEBUG ] 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691503" 22:25:03 INFO [http_cmd]: http status code OK 22:25:03 DEBUG --- status code --- 22:25:03 DEBUG http status code is 200 (expected 200) 22:25:03 DEBUG --- http response --- 22:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:03 DEBUG ----- output ----- 22:25:03 DEBUG { 22:25:03 DEBUG "status": "success", 22:25:03 DEBUG "data": { 22:25:03 DEBUG "resultType": "vector", 22:25:03 DEBUG "result": [ 22:25:03 DEBUG { 22:25:03 DEBUG "metric": { 22:25:03 DEBUG "instance": "10.142.0.100:9100" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG [...] 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG 22:25:03 DEBUG "instance": "10.142.0.99:9100" 22:25:03 DEBUG }, 22:25:03 DEBUG "value": [ 22:25:03 DEBUG 1686691503, 22:25:03 DEBUG "21283687.46666667" 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG ] 22:25:03 DEBUG } 22:25:03 DEBUG } 22:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691504" 22:25:04 INFO [http_cmd]: http status code OK 22:25:04 DEBUG --- status code --- 22:25:04 DEBUG http status code is 200 (expected 200) 22:25:04 DEBUG --- http response --- 22:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:04 DEBUG ----- output ----- 22:25:04 DEBUG { 22:25:04 DEBUG "status": "success", 22:25:04 DEBUG "data": { 22:25:04 DEBUG "resultType": "vector", 22:25:04 DEBUG "result": [ 22:25:04 DEBUG { 22:25:04 DEBUG "metric": { 22:25:04 DEBUG "pod": "ds-idrepo-2" 22:25:04 DEBUG }, 22:25:04 DEBUG "value": [ 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG [...] 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG "pod": "ds-idrepo-0" 22:25:04 DEBUG }, 22:25:04 DEBUG "value": [ 22:25:04 DEBUG 1686691504, 22:25:04 DEBUG "661.1" 22:25:04 DEBUG ] 22:25:04 DEBUG } 22:25:04 DEBUG ] 22:25:04 DEBUG } 22:25:04 DEBUG } 22:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691504" 22:25:04 INFO [http_cmd]: http status code OK 22:25:04 DEBUG --- status code --- 22:25:04 DEBUG http status code is 200 (expected 200) 22:25:04 DEBUG --- http response --- 22:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:25:04 DEBUG ----- output ----- 22:25:04 DEBUG { 22:25:04 DEBUG "status": "success", 22:25:04 DEBUG "data": { 22:25:04 DEBUG "resultType": "vector", 22:25:04 DEBUG "result": [ 22:25:04 DEBUG { 22:25:04 DEBUG "metric": { 22:25:04 DEBUG "pod": "ds-cts-2" 22:25:04 DEBUG }, 22:25:04 DEBUG "value": [ 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG [...] 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:25:04 DEBUG }, 22:25:04 DEBUG "value": [ 22:25:04 DEBUG 1686691504, 22:25:04 DEBUG "0" 22:25:04 DEBUG ] 22:25:04 DEBUG } 22:25:04 DEBUG ] 22:25:04 DEBUG } 22:25:04 DEBUG } 22:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691504" 22:25:04 INFO [http_cmd]: http status code OK 22:25:04 DEBUG --- status code --- 22:25:04 DEBUG http status code is 200 (expected 200) 22:25:04 DEBUG --- http response --- 22:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:04 DEBUG ----- output ----- 22:25:04 DEBUG { 22:25:04 DEBUG "status": "success", 22:25:04 DEBUG "data": { 22:25:04 DEBUG "resultType": "vector", 22:25:04 DEBUG "result": [ 22:25:04 DEBUG { 22:25:04 DEBUG "metric": { 22:25:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:25:04 DEBUG }, 22:25:04 DEBUG "value": [ 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG [...] 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG 22:25:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:25:04 DEBUG }, 22:25:04 DEBUG "value": [ 22:25:04 DEBUG 1686691504, 22:25:04 DEBUG "0.25100229325974727" 22:25:04 DEBUG ] 22:25:04 DEBUG } 22:25:04 DEBUG ] 22:25:04 DEBUG } 22:25:04 DEBUG } 22:25:20 INFO 22:25:20 INFO [loop_until]: kubectl --namespace=xlou top pods 22:25:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:25:20 INFO [loop_until]: OK (rc = 0) 22:25:20 DEBUG --- stdout --- 22:25:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 5m 5757Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 10m 417Mi ds-cts-1 10m 425Mi ds-cts-2 10m 393Mi ds-idrepo-0 3434m 13808Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3467m 4927Mi idm-58dc667486-rgz9j 2787m 5096Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 214m 641Mi 22:25:20 DEBUG --- stderr --- 22:25:20 DEBUG 22:25:24 INFO 22:25:24 INFO [loop_until]: kubectl --namespace=xlou top node 22:25:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:25:24 INFO [loop_until]: OK (rc = 0) 22:25:24 DEBUG --- stdout --- 22:25:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3415m 21% 5871Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 65m 0% 6856Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6828Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2574m 16% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 430m 2% 3128Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1171Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 65m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13718Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3588m 22% 14508Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14598Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 278m 1% 1958Mi 3% 22:25:24 DEBUG --- stderr --- 22:25:24 DEBUG 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "0" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "0" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG { 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG "data": { 22:26:03 DEBUG { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG "data": { 22:26:03 DEBUG { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "result": [ 22:26:03 DEBUG "pod": "ds-cts-0" 22:26:03 DEBUG { 22:26:03 DEBUG }, 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "value": [ 22:26:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:26:03 DEBUG 22:26:03 DEBUG }, 22:26:03 DEBUG 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "ds-idrepo-0" 22:26:03 DEBUG 22:26:03 DEBUG }, 22:26:03 DEBUG 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG "value": [ 22:26:03 DEBUG "pod": "ds-idrepo-0" 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG }, 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG "0" 22:26:03 DEBUG "value": [ 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG ] 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:26:03 DEBUG } 22:26:03 DEBUG "3.3388817966019535" 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG ] 22:26:03 DEBUG ] 22:26:03 DEBUG { 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG "data": { 22:26:03 DEBUG } 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG } 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "ds-idrepo-0" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "494435638.9016922" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "ds-idrepo-2" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "ds-idrepo-0" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "596.7666666666667" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "ds-cts-0" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "ds-idrepo-0" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "596.7666666666667" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "ds-cts-0" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "ds-cts-2" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "0" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691563" 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "0" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "0.2" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "instance": "10.142.0.100:9100" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "instance": "10.142.0.99:9100" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "0" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "ds-idrepo-0" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "6778.659379732666" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "13617123328" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "0" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "pod": "ds-idrepo-2" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "pod": "ds-idrepo-0" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "596.7666666666667" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "instance": "10.142.0.100:9100" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "instance": "10.142.0.99:9100" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "0.0003333333333330302" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691563" 22:26:03 INFO [http_cmd]: http status code OK 22:26:03 DEBUG --- status code --- 22:26:03 DEBUG http status code is 200 (expected 200) 22:26:03 DEBUG --- http response --- 22:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:03 DEBUG ----- output ----- 22:26:03 DEBUG { 22:26:03 DEBUG "status": "success", 22:26:03 DEBUG "data": { 22:26:03 DEBUG "resultType": "vector", 22:26:03 DEBUG "result": [ 22:26:03 DEBUG { 22:26:03 DEBUG "metric": { 22:26:03 DEBUG "instance": "10.142.0.100:9100" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG [...] 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG 22:26:03 DEBUG "instance": "10.142.0.99:9100" 22:26:03 DEBUG }, 22:26:03 DEBUG "value": [ 22:26:03 DEBUG 1686691563, 22:26:03 DEBUG "18434306.4" 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG ] 22:26:03 DEBUG } 22:26:03 DEBUG } 22:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691563" 22:26:04 INFO [http_cmd]: http status code OK 22:26:04 DEBUG --- status code --- 22:26:04 DEBUG http status code is 200 (expected 200) 22:26:04 DEBUG --- http response --- 22:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:04 DEBUG ----- output ----- 22:26:04 DEBUG { 22:26:04 DEBUG "status": "success", 22:26:04 DEBUG "data": { 22:26:04 DEBUG "resultType": "vector", 22:26:04 DEBUG "result": [ 22:26:04 DEBUG { 22:26:04 DEBUG "metric": { 22:26:04 DEBUG "instance": "10.142.0.100:9100" 22:26:04 DEBUG }, 22:26:04 DEBUG "value": [ 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG [...] 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG "instance": "10.142.0.99:9100" 22:26:04 DEBUG }, 22:26:04 DEBUG "value": [ 22:26:04 DEBUG 1686691563, 22:26:04 DEBUG "18430109.3" 22:26:04 DEBUG ] 22:26:04 DEBUG } 22:26:04 DEBUG ] 22:26:04 DEBUG } 22:26:04 DEBUG } 22:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691564" 22:26:04 INFO [http_cmd]: http status code OK 22:26:04 DEBUG --- status code --- 22:26:04 DEBUG http status code is 200 (expected 200) 22:26:04 DEBUG --- http response --- 22:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:04 DEBUG ----- output ----- 22:26:04 DEBUG { 22:26:04 DEBUG "status": "success", 22:26:04 DEBUG "data": { 22:26:04 DEBUG "resultType": "vector", 22:26:04 DEBUG "result": [ 22:26:04 DEBUG { 22:26:04 DEBUG "metric": { 22:26:04 DEBUG "pod": "ds-idrepo-2" 22:26:04 DEBUG }, 22:26:04 DEBUG "value": [ 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG [...] 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG "pod": "ds-idrepo-0" 22:26:04 DEBUG }, 22:26:04 DEBUG "value": [ 22:26:04 DEBUG 1686691564, 22:26:04 DEBUG "596.7666666666667" 22:26:04 DEBUG ] 22:26:04 DEBUG } 22:26:04 DEBUG ] 22:26:04 DEBUG } 22:26:04 DEBUG } 22:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691564" 22:26:04 INFO [http_cmd]: http status code OK 22:26:04 DEBUG --- status code --- 22:26:04 DEBUG http status code is 200 (expected 200) 22:26:04 DEBUG --- http response --- 22:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:26:04 DEBUG ----- output ----- 22:26:04 DEBUG { 22:26:04 DEBUG "status": "success", 22:26:04 DEBUG "data": { 22:26:04 DEBUG "resultType": "vector", 22:26:04 DEBUG "result": [ 22:26:04 DEBUG { 22:26:04 DEBUG "metric": { 22:26:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:26:04 DEBUG }, 22:26:04 DEBUG "value": [ 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG [...] 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG "pod": "ds-idrepo-0" 22:26:04 DEBUG }, 22:26:04 DEBUG "value": [ 22:26:04 DEBUG 1686691564, 22:26:04 DEBUG "0.7901494699413972" 22:26:04 DEBUG ] 22:26:04 DEBUG } 22:26:04 DEBUG ] 22:26:04 DEBUG } 22:26:04 DEBUG } 22:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691564" 22:26:04 INFO [http_cmd]: http status code OK 22:26:04 DEBUG --- status code --- 22:26:04 DEBUG http status code is 200 (expected 200) 22:26:04 DEBUG --- http response --- 22:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:04 DEBUG ----- output ----- 22:26:04 DEBUG { 22:26:04 DEBUG "status": "success", 22:26:04 DEBUG "data": { 22:26:04 DEBUG "resultType": "vector", 22:26:04 DEBUG "result": [ 22:26:04 DEBUG { 22:26:04 DEBUG "metric": { 22:26:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:26:04 DEBUG }, 22:26:04 DEBUG "value": [ 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG [...] 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG 22:26:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:26:04 DEBUG }, 22:26:04 DEBUG "value": [ 22:26:04 DEBUG 1686691564, 22:26:04 DEBUG "0.2373546029768831" 22:26:04 DEBUG ] 22:26:04 DEBUG } 22:26:04 DEBUG ] 22:26:04 DEBUG } 22:26:04 DEBUG } 22:26:20 INFO 22:26:20 INFO [loop_until]: kubectl --namespace=xlou top pods 22:26:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:26:20 INFO [loop_until]: OK (rc = 0) 22:26:20 DEBUG --- stdout --- 22:26:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5757Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 392Mi ds-idrepo-0 3414m 13828Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2783m 4928Mi idm-58dc667486-rgz9j 2946m 5096Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 208m 641Mi 22:26:20 DEBUG --- stderr --- 22:26:20 DEBUG 22:26:24 INFO 22:26:24 INFO [loop_until]: kubectl --namespace=xlou top node 22:26:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:26:24 INFO [loop_until]: OK (rc = 0) 22:26:24 DEBUG --- stdout --- 22:26:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2902m 18% 5874Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 65m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6831Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2797m 17% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6946Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 444m 2% 3134Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 65m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 66m 0% 13723Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3403m 21% 14499Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14604Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 274m 1% 1959Mi 3% 22:26:24 DEBUG --- stderr --- 22:26:24 DEBUG 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "ds-idrepo-2" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "ds-idrepo-0" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "616" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "ds-cts-0" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "ds-idrepo-0" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "ds-cts-0" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "ds-cts-2" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "ds-cts-0" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "ds-idrepo-0" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "615.9666666666667" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691623" 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0.2" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "instance": "10.142.0.100:9100" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "instance": "10.142.0.99:9100" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "13616017408" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "pod": "ds-idrepo-2" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "pod": "ds-idrepo-0" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "616" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691623" 22:27:03 INFO [http_cmd]: http status code OK 22:27:03 DEBUG --- status code --- 22:27:03 DEBUG http status code is 200 (expected 200) 22:27:03 DEBUG --- http response --- 22:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:03 DEBUG ----- output ----- 22:27:03 DEBUG { 22:27:03 DEBUG "status": "success", 22:27:03 DEBUG "data": { 22:27:03 DEBUG "resultType": "vector", 22:27:03 DEBUG "result": [ 22:27:03 DEBUG { 22:27:03 DEBUG "metric": { 22:27:03 DEBUG "instance": "10.142.0.100:9100" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG [...] 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG 22:27:03 DEBUG "instance": "10.142.0.99:9100" 22:27:03 DEBUG }, 22:27:03 DEBUG "value": [ 22:27:03 DEBUG 1686691623, 22:27:03 DEBUG "0.00026666666666604516" 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG ] 22:27:03 DEBUG } 22:27:03 DEBUG } 22:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691624" 22:27:04 INFO [http_cmd]: http status code OK 22:27:04 DEBUG --- status code --- 22:27:04 DEBUG http status code is 200 (expected 200) 22:27:04 DEBUG --- http response --- 22:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:04 DEBUG ----- output ----- 22:27:04 DEBUG { 22:27:04 DEBUG "status": "success", 22:27:04 DEBUG "data": { 22:27:04 DEBUG "resultType": "vector", 22:27:04 DEBUG "result": [ 22:27:04 DEBUG { 22:27:04 DEBUG "metric": { 22:27:04 DEBUG "instance": "10.142.0.100:9100" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG [...] 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG "instance": "10.142.0.99:9100" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 1686691624, 22:27:04 DEBUG "19230870.6" 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG } 22:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691624" 22:27:04 INFO [http_cmd]: http status code OK 22:27:04 DEBUG --- status code --- 22:27:04 DEBUG http status code is 200 (expected 200) 22:27:04 DEBUG --- http response --- 22:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:04 DEBUG ----- output ----- 22:27:04 DEBUG { 22:27:04 DEBUG "status": "success", 22:27:04 DEBUG "data": { 22:27:04 DEBUG "resultType": "vector", 22:27:04 DEBUG "result": [ 22:27:04 DEBUG { 22:27:04 DEBUG "metric": { 22:27:04 DEBUG "instance": "10.142.0.100:9100" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG [...] 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG "instance": "10.142.0.99:9100" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 1686691624, 22:27:04 DEBUG "19226847.4" 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG } 22:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691624" 22:27:04 INFO [http_cmd]: http status code OK 22:27:04 DEBUG --- status code --- 22:27:04 DEBUG http status code is 200 (expected 200) 22:27:04 DEBUG --- http response --- 22:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:04 DEBUG ----- output ----- 22:27:04 DEBUG { 22:27:04 DEBUG "status": "success", 22:27:04 DEBUG "data": { 22:27:04 DEBUG "resultType": "vector", 22:27:04 DEBUG "result": [ 22:27:04 DEBUG { 22:27:04 DEBUG "metric": { 22:27:04 DEBUG "pod": "ds-idrepo-2" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG [...] 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG "pod": "ds-idrepo-0" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 1686691624, 22:27:04 DEBUG "616.0333333333333" 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG } 22:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691624" 22:27:04 INFO [http_cmd]: http status code OK 22:27:04 DEBUG --- status code --- 22:27:04 DEBUG http status code is 200 (expected 200) 22:27:04 DEBUG --- http response --- 22:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:27:04 DEBUG ----- output ----- 22:27:04 DEBUG { 22:27:04 DEBUG "status": "success", 22:27:04 DEBUG "data": { 22:27:04 DEBUG "resultType": "vector", 22:27:04 DEBUG "result": [ 22:27:04 DEBUG { 22:27:04 DEBUG "metric": { 22:27:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG [...] 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 1686691624, 22:27:04 DEBUG "0" 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG } 22:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691624" 22:27:04 INFO [http_cmd]: http status code OK 22:27:04 DEBUG --- status code --- 22:27:04 DEBUG http status code is 200 (expected 200) 22:27:04 DEBUG --- http response --- 22:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:04 DEBUG ----- output ----- 22:27:04 DEBUG { 22:27:04 DEBUG "status": "success", 22:27:04 DEBUG "data": { 22:27:04 DEBUG "resultType": "vector", 22:27:04 DEBUG "result": [ 22:27:04 DEBUG { 22:27:04 DEBUG "metric": { 22:27:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG [...] 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG 22:27:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:27:04 DEBUG }, 22:27:04 DEBUG "value": [ 22:27:04 DEBUG 1686691624, 22:27:04 DEBUG "0.2315432427942451" 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG ] 22:27:04 DEBUG } 22:27:04 DEBUG } 22:27:20 INFO 22:27:20 INFO [loop_until]: kubectl --namespace=xlou top pods 22:27:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:27:20 INFO [loop_until]: OK (rc = 0) 22:27:20 DEBUG --- stdout --- 22:27:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5757Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3358m 13817Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3016m 4930Mi idm-58dc667486-rgz9j 2766m 5096Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 212m 641Mi 22:27:20 DEBUG --- stderr --- 22:27:20 DEBUG 22:27:24 INFO 22:27:24 INFO [loop_until]: kubectl --namespace=xlou top node 22:27:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:27:24 INFO [loop_until]: OK (rc = 0) 22:27:24 DEBUG --- stdout --- 22:27:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3179m 20% 5873Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6860Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6825Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2777m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 443m 2% 3130Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1482Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13721Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3426m 21% 14507Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14599Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 282m 1% 1958Mi 3% 22:27:24 DEBUG --- stderr --- 22:27:24 DEBUG 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "ds-cts-2" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "0.00003196291542357623" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "ds-idrepo-2" 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "ds-cts-2" 22:28:03 DEBUG }, 22:28:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "pod": "ds-idrepo-0" 22:28:03 DEBUG 22:28:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:28:03 DEBUG }, 22:28:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "675.7666666666667" 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "0" 22:28:03 DEBUG ] 22:28:03 DEBUG "0" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "0" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:28:03 DEBUG { 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG { 22:28:03 DEBUG "data": { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "result": [ 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG { 22:28:03 DEBUG "result": [ 22:28:03 DEBUG "metric": { 22:28:03 DEBUG { 22:28:03 DEBUG "pod": "ds-cts-0" 22:28:03 DEBUG "metric": { 22:28:03 DEBUG }, 22:28:03 DEBUG "pod": "ds-cts-0" 22:28:03 DEBUG "value": [ 22:28:03 DEBUG }, 22:28:03 DEBUG 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "pod": "ds-idrepo-0" 22:28:03 DEBUG 22:28:03 DEBUG }, 22:28:03 DEBUG "pod": "ds-idrepo-0" 22:28:03 DEBUG "value": [ 22:28:03 DEBUG }, 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG "0" 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG ] 22:28:03 DEBUG "675.7333333333333" 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "0" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "ds-cts-0" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "pod": "ds-cts-2" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "0" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691683" 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "0.2" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "ds-cts-2" 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG }, 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG "value": [ 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:03 DEBUG 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG [...] 22:28:03 DEBUG { 22:28:03 DEBUG 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG 22:28:03 DEBUG "data": { 22:28:03 DEBUG 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:28:03 DEBUG "result": [ 22:28:03 DEBUG }, 22:28:03 DEBUG { 22:28:03 DEBUG "value": [ 22:28:03 DEBUG "metric": { 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "instance": "10.142.0.100:9100" 22:28:03 DEBUG "0" 22:28:03 DEBUG }, 22:28:03 DEBUG ] 22:28:03 DEBUG "value": [ 22:28:03 DEBUG } 22:28:03 DEBUG 22:28:03 DEBUG ] 22:28:03 DEBUG 22:28:03 DEBUG } 22:28:03 DEBUG 22:28:03 DEBUG } 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "instance": "10.142.0.99:9100" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "0" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691683" 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "13616549888" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "0" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691683" 22:28:03 INFO [http_cmd]: http status code OK 22:28:03 DEBUG --- status code --- 22:28:03 DEBUG http status code is 200 (expected 200) 22:28:03 DEBUG --- http response --- 22:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:03 DEBUG ----- output ----- 22:28:03 DEBUG { 22:28:03 DEBUG "status": "success", 22:28:03 DEBUG "data": { 22:28:03 DEBUG "resultType": "vector", 22:28:03 DEBUG "result": [ 22:28:03 DEBUG { 22:28:03 DEBUG "metric": { 22:28:03 DEBUG "pod": "ds-idrepo-2" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG [...] 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG 22:28:03 DEBUG "pod": "ds-idrepo-0" 22:28:03 DEBUG }, 22:28:03 DEBUG "value": [ 22:28:03 DEBUG 1686691683, 22:28:03 DEBUG "675.7666666666667" 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG ] 22:28:03 DEBUG } 22:28:03 DEBUG } 22:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691684" 22:28:04 INFO [http_cmd]: http status code OK 22:28:04 DEBUG --- status code --- 22:28:04 DEBUG http status code is 200 (expected 200) 22:28:04 DEBUG --- http response --- 22:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:04 DEBUG ----- output ----- 22:28:04 DEBUG { 22:28:04 DEBUG "status": "success", 22:28:04 DEBUG "data": { 22:28:04 DEBUG "resultType": "vector", 22:28:04 DEBUG "result": [ 22:28:04 DEBUG { 22:28:04 DEBUG "metric": { 22:28:04 DEBUG "instance": "10.142.0.100:9100" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG [...] 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG "instance": "10.142.0.99:9100" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 1686691684, 22:28:04 DEBUG "0.00030000000000048505" 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG } 22:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691684" 22:28:04 INFO [http_cmd]: http status code OK 22:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691684" 22:28:04 DEBUG --- status code --- 22:28:04 DEBUG http status code is 200 (expected 200) 22:28:04 DEBUG --- http response --- 22:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:04 DEBUG ----- output ----- 22:28:04 DEBUG { 22:28:04 DEBUG "status": "success", 22:28:04 DEBUG "data": { 22:28:04 DEBUG "resultType": "vector", 22:28:04 DEBUG "result": [ 22:28:04 DEBUG { 22:28:04 DEBUG "metric": { 22:28:04 DEBUG "instance": "10.142.0.100:9100" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG [...] 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG "instance": "10.142.0.99:9100" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 1686691684, 22:28:04 DEBUG "21795496.666666664" 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG } 22:28:04 INFO [http_cmd]: http status code OK 22:28:04 DEBUG --- status code --- 22:28:04 DEBUG http status code is 200 (expected 200) 22:28:04 DEBUG --- http response --- 22:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:04 DEBUG ----- output ----- 22:28:04 DEBUG { 22:28:04 DEBUG "status": "success", 22:28:04 DEBUG "data": { 22:28:04 DEBUG "resultType": "vector", 22:28:04 DEBUG "result": [ 22:28:04 DEBUG { 22:28:04 DEBUG "metric": { 22:28:04 DEBUG "instance": "10.142.0.100:9100" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG [...] 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG "instance": "10.142.0.99:9100" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 1686691684, 22:28:04 DEBUG "21799483.066666666" 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG } 22:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691684" 22:28:04 INFO [http_cmd]: http status code OK 22:28:04 DEBUG --- status code --- 22:28:04 DEBUG http status code is 200 (expected 200) 22:28:04 DEBUG --- http response --- 22:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:04 DEBUG ----- output ----- 22:28:04 DEBUG { 22:28:04 DEBUG "status": "success", 22:28:04 DEBUG "data": { 22:28:04 DEBUG "resultType": "vector", 22:28:04 DEBUG "result": [ 22:28:04 DEBUG { 22:28:04 DEBUG "metric": { 22:28:04 DEBUG "pod": "ds-idrepo-2" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG [...] 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG "pod": "ds-idrepo-0" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 1686691684, 22:28:04 DEBUG "675.7666666666667" 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG } 22:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691684" 22:28:04 INFO [http_cmd]: http status code OK 22:28:04 DEBUG --- status code --- 22:28:04 DEBUG http status code is 200 (expected 200) 22:28:04 DEBUG --- http response --- 22:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:28:04 DEBUG ----- output ----- 22:28:04 DEBUG { 22:28:04 DEBUG "status": "success", 22:28:04 DEBUG "data": { 22:28:04 DEBUG "resultType": "vector", 22:28:04 DEBUG "result": [ 22:28:04 DEBUG { 22:28:04 DEBUG "metric": { 22:28:04 DEBUG "pod": "ds-cts-2" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG [...] 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 1686691684, 22:28:04 DEBUG "0" 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG } 22:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691684" 22:28:04 INFO [http_cmd]: http status code OK 22:28:04 DEBUG --- status code --- 22:28:04 DEBUG http status code is 200 (expected 200) 22:28:04 DEBUG --- http response --- 22:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:04 DEBUG ----- output ----- 22:28:04 DEBUG { 22:28:04 DEBUG "status": "success", 22:28:04 DEBUG "data": { 22:28:04 DEBUG "resultType": "vector", 22:28:04 DEBUG "result": [ 22:28:04 DEBUG { 22:28:04 DEBUG "metric": { 22:28:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG [...] 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG 22:28:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:28:04 DEBUG }, 22:28:04 DEBUG "value": [ 22:28:04 DEBUG 1686691684, 22:28:04 DEBUG "0.24634458076196514" 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG ] 22:28:04 DEBUG } 22:28:04 DEBUG } 22:28:21 INFO 22:28:21 INFO [loop_until]: kubectl --namespace=xlou top pods 22:28:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:28:21 INFO [loop_until]: OK (rc = 0) 22:28:21 DEBUG --- stdout --- 22:28:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5757Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3391m 13800Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3327m 4931Mi idm-58dc667486-rgz9j 2574m 5097Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 208m 642Mi 22:28:21 DEBUG --- stderr --- 22:28:21 DEBUG 22:28:24 INFO 22:28:24 INFO [loop_until]: kubectl --namespace=xlou top node 22:28:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:28:24 INFO [loop_until]: OK (rc = 0) 22:28:24 DEBUG --- stdout --- 22:28:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3502m 22% 5872Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6831Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2609m 16% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 439m 2% 3133Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1485Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1171Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 67m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13724Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3566m 22% 14528Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14600Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 272m 1% 1956Mi 3% 22:28:24 DEBUG --- stderr --- 22:28:24 DEBUG 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:03 DEBUG { 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG { 22:29:03 DEBUG "data": { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "result": [ 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG { 22:29:03 DEBUG "result": [ 22:29:03 DEBUG "metric": { 22:29:03 DEBUG { 22:29:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:29:03 DEBUG "metric": { 22:29:03 DEBUG }, 22:29:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:29:03 DEBUG "value": [ 22:29:03 DEBUG }, 22:29:03 DEBUG 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:29:03 DEBUG 22:29:03 DEBUG }, 22:29:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:29:03 DEBUG "value": [ 22:29:03 DEBUG }, 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG "0" 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG ] 22:29:03 DEBUG "0.004464498352523325" 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG { 22:29:03 DEBUG { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "data": { 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "result": [ 22:29:03 DEBUG "result": [ 22:29:03 DEBUG { 22:29:03 DEBUG { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "pod": "ds-cts-0" 22:29:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:29:03 DEBUG }, 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG "pod": "ds-idrepo-0" 22:29:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:29:03 DEBUG }, 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "0" 22:29:03 DEBUG "0" 22:29:03 DEBUG ] 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG { 22:29:03 DEBUG { 22:29:03 DEBUG { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "data": { 22:29:03 DEBUG "data": { 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "result": [ 22:29:03 DEBUG "result": [ 22:29:03 DEBUG "result": [ 22:29:03 DEBUG { 22:29:03 DEBUG { 22:29:03 DEBUG { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:29:03 DEBUG "pod": "ds-idrepo-2" 22:29:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:29:03 DEBUG }, 22:29:03 DEBUG }, 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG "value": [ 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG [...] 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:29:03 DEBUG "pod": "ds-idrepo-0" 22:29:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:29:03 DEBUG }, 22:29:03 DEBUG }, 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG "value": [ 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "0" 22:29:03 DEBUG "591.3666666666667" 22:29:03 DEBUG "158443534.40557852" 22:29:03 DEBUG ] 22:29:03 DEBUG ] 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG ] 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "result": [ 22:29:03 DEBUG { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "pod": "ds-cts-0" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG "pod": "ds-idrepo-0" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "591.3666666666667" 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "result": [ 22:29:03 DEBUG { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "pod": "ds-cts-0" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG "pod": "ds-cts-2" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "0" 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "result": [ 22:29:03 DEBUG { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "0.2" 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691743" 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "result": [ 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG { 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG "metric": { 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG }, 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:03 DEBUG "value": [ 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG 22:29:03 DEBUG { 22:29:03 DEBUG 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG 22:29:03 DEBUG "data": { 22:29:03 DEBUG [...] 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG 22:29:03 DEBUG "result": [ 22:29:03 DEBUG 22:29:03 DEBUG { 22:29:03 DEBUG 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:29:03 DEBUG "instance": "10.142.0.100:9100" 22:29:03 DEBUG }, 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG 22:29:03 DEBUG "0" 22:29:03 DEBUG 22:29:03 DEBUG ] 22:29:03 DEBUG 22:29:03 DEBUG } 22:29:03 DEBUG [...] 22:29:03 DEBUG ] 22:29:03 DEBUG 22:29:03 DEBUG } 22:29:03 DEBUG 22:29:03 DEBUG } 22:29:03 DEBUG 22:29:03 DEBUG "instance": "10.142.0.99:9100" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "0" 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691743" 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "result": [ 22:29:03 DEBUG { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "0" 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "result": [ 22:29:03 DEBUG { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "13619015680" 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691743" 22:29:03 INFO [http_cmd]: http status code OK 22:29:03 DEBUG --- status code --- 22:29:03 DEBUG http status code is 200 (expected 200) 22:29:03 DEBUG --- http response --- 22:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:03 DEBUG ----- output ----- 22:29:03 DEBUG { 22:29:03 DEBUG "status": "success", 22:29:03 DEBUG "data": { 22:29:03 DEBUG "resultType": "vector", 22:29:03 DEBUG "result": [ 22:29:03 DEBUG { 22:29:03 DEBUG "metric": { 22:29:03 DEBUG "pod": "ds-idrepo-2" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG [...] 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG 22:29:03 DEBUG "pod": "ds-idrepo-0" 22:29:03 DEBUG }, 22:29:03 DEBUG "value": [ 22:29:03 DEBUG 1686691743, 22:29:03 DEBUG "591.3333333333334" 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG ] 22:29:03 DEBUG } 22:29:03 DEBUG } 22:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691744" 22:29:04 INFO [http_cmd]: http status code OK 22:29:04 DEBUG --- status code --- 22:29:04 DEBUG http status code is 200 (expected 200) 22:29:04 DEBUG --- http response --- 22:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:04 DEBUG ----- output ----- 22:29:04 DEBUG { 22:29:04 DEBUG "status": "success", 22:29:04 DEBUG "data": { 22:29:04 DEBUG "resultType": "vector", 22:29:04 DEBUG "result": [ 22:29:04 DEBUG { 22:29:04 DEBUG "metric": { 22:29:04 DEBUG "instance": "10.142.0.100:9100" 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG [...] 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG "instance": "10.142.0.99:9100" 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 1686691744, 22:29:04 DEBUG "0.0003333333333330302" 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG } 22:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691744" 22:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691744" 22:29:04 INFO [http_cmd]: http status code OK 22:29:04 INFO [http_cmd]: http status code OK 22:29:04 DEBUG --- status code --- 22:29:04 DEBUG --- status code --- 22:29:04 DEBUG http status code is 200 (expected 200) 22:29:04 DEBUG http status code is 200 (expected 200) 22:29:04 DEBUG --- http response --- 22:29:04 DEBUG --- http response --- 22:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:04 DEBUG ----- output ----- 22:29:04 DEBUG ----- output ----- 22:29:04 DEBUG { 22:29:04 DEBUG { 22:29:04 DEBUG "status": "success", 22:29:04 DEBUG "status": "success", 22:29:04 DEBUG "data": { 22:29:04 DEBUG "data": { 22:29:04 DEBUG "resultType": "vector", 22:29:04 DEBUG "resultType": "vector", 22:29:04 DEBUG "result": [ 22:29:04 DEBUG "result": [ 22:29:04 DEBUG { 22:29:04 DEBUG { 22:29:04 DEBUG "metric": { 22:29:04 DEBUG "metric": { 22:29:04 DEBUG "instance": "10.142.0.100:9100" 22:29:04 DEBUG "instance": "10.142.0.100:9100" 22:29:04 DEBUG }, 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG [...] 22:29:04 DEBUG [...] 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG "instance": "10.142.0.99:9100" 22:29:04 DEBUG "instance": "10.142.0.99:9100" 22:29:04 DEBUG }, 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 1686691744, 22:29:04 DEBUG 1686691744, 22:29:04 DEBUG "19626273.6" 22:29:04 DEBUG "19621986" 22:29:04 DEBUG ] 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG } 22:29:04 DEBUG ] 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG } 22:29:04 DEBUG } 22:29:04 DEBUG } 22:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691744" 22:29:04 INFO [http_cmd]: http status code OK 22:29:04 DEBUG --- status code --- 22:29:04 DEBUG http status code is 200 (expected 200) 22:29:04 DEBUG --- http response --- 22:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:04 DEBUG ----- output ----- 22:29:04 DEBUG { 22:29:04 DEBUG "status": "success", 22:29:04 DEBUG "data": { 22:29:04 DEBUG "resultType": "vector", 22:29:04 DEBUG "result": [ 22:29:04 DEBUG { 22:29:04 DEBUG "metric": { 22:29:04 DEBUG "pod": "ds-idrepo-2" 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG [...] 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG "pod": "ds-idrepo-0" 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 1686691744, 22:29:04 DEBUG "591.3333333333334" 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG } 22:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691744" 22:29:04 INFO [http_cmd]: http status code OK 22:29:04 DEBUG --- status code --- 22:29:04 DEBUG http status code is 200 (expected 200) 22:29:04 DEBUG --- http response --- 22:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:04 DEBUG ----- output ----- 22:29:04 DEBUG { 22:29:04 DEBUG "status": "success", 22:29:04 DEBUG "data": { 22:29:04 DEBUG "resultType": "vector", 22:29:04 DEBUG "result": [ 22:29:04 DEBUG { 22:29:04 DEBUG "metric": { 22:29:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG [...] 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 1686691744, 22:29:04 DEBUG "0" 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG } 22:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691744" 22:29:04 INFO [http_cmd]: http status code OK 22:29:04 DEBUG --- status code --- 22:29:04 DEBUG http status code is 200 (expected 200) 22:29:04 DEBUG --- http response --- 22:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:04 DEBUG ----- output ----- 22:29:04 DEBUG { 22:29:04 DEBUG "status": "success", 22:29:04 DEBUG "data": { 22:29:04 DEBUG "resultType": "vector", 22:29:04 DEBUG "result": [ 22:29:04 DEBUG { 22:29:04 DEBUG "metric": { 22:29:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG [...] 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG 22:29:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:29:04 DEBUG }, 22:29:04 DEBUG "value": [ 22:29:04 DEBUG 1686691744, 22:29:04 DEBUG "0.22494908591019883" 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG ] 22:29:04 DEBUG } 22:29:04 DEBUG } 22:29:21 INFO 22:29:21 INFO [loop_until]: kubectl --namespace=xlou top pods 22:29:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:29:21 INFO [loop_until]: OK (rc = 0) 22:29:21 DEBUG --- stdout --- 22:29:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5757Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3135m 13800Mi ds-idrepo-1 20m 13821Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3032m 4933Mi idm-58dc667486-rgz9j 2550m 5097Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 199m 641Mi 22:29:21 DEBUG --- stderr --- 22:29:21 DEBUG 22:29:24 INFO 22:29:24 INFO [loop_until]: kubectl --namespace=xlou top node 22:29:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:29:24 INFO [loop_until]: OK (rc = 0) 22:29:24 DEBUG --- stdout --- 22:29:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3019m 18% 5873Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6861Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2680m 16% 6075Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 441m 2% 3134Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 85m 0% 1483Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 65m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13722Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3206m 20% 14510Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 75m 0% 14603Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 264m 1% 1959Mi 3% 22:29:24 DEBUG --- stderr --- 22:29:24 DEBUG 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "0" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "0" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "ds-idrepo-2" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "ds-idrepo-0" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "647.3333333333334" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "0" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "ds-cts-0" 22:30:03 DEBUG "pod": "ds-cts-0" 22:30:03 DEBUG }, 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "ds-idrepo-0" 22:30:03 DEBUG "pod": "ds-idrepo-0" 22:30:03 DEBUG }, 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "647.3" 22:30:03 DEBUG "0" 22:30:03 DEBUG ] 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG }, 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:30:03 DEBUG "value": [ 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG { 22:30:03 DEBUG "0" 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG ] 22:30:03 DEBUG "data": { 22:30:03 DEBUG } 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG ] 22:30:03 DEBUG "result": [ 22:30:03 DEBUG } 22:30:03 DEBUG { 22:30:03 DEBUG } 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "0" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "0.2" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "ds-cts-0" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "ds-cts-2" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "0" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "lodemon-6455d5ff7c-zzndq" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "0" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "instance": "10.142.0.100:9100" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "instance": "10.142.0.99:9100" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "0" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "0" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691803" 22:30:03 INFO [http_cmd]: http status code OK 22:30:03 DEBUG --- status code --- 22:30:03 DEBUG http status code is 200 (expected 200) 22:30:03 DEBUG --- http response --- 22:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:03 DEBUG ----- output ----- 22:30:03 DEBUG { 22:30:03 DEBUG "status": "success", 22:30:03 DEBUG "data": { 22:30:03 DEBUG "resultType": "vector", 22:30:03 DEBUG "result": [ 22:30:03 DEBUG { 22:30:03 DEBUG "metric": { 22:30:03 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG [...] 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG 22:30:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:30:03 DEBUG }, 22:30:03 DEBUG "value": [ 22:30:03 DEBUG 1686691803, 22:30:03 DEBUG "13615894528" 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG ] 22:30:03 DEBUG } 22:30:03 DEBUG } 22:30:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691804" 22:30:04 INFO [http_cmd]: http status code OK 22:30:04 DEBUG --- status code --- 22:30:04 DEBUG http status code is 200 (expected 200) 22:30:04 DEBUG --- http response --- 22:30:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:04 DEBUG ----- output ----- 22:30:04 DEBUG { 22:30:04 DEBUG "status": "success", 22:30:04 DEBUG "data": { 22:30:04 DEBUG "resultType": "vector", 22:30:04 DEBUG "result": [ 22:30:04 DEBUG { 22:30:04 DEBUG "metric": { 22:30:04 DEBUG "pod": "ds-idrepo-2" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG [...] 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG "pod": "ds-idrepo-0" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 1686691804, 22:30:04 DEBUG "647.3" 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG } 22:30:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691804" 22:30:04 INFO [http_cmd]: http status code OK 22:30:04 DEBUG --- status code --- 22:30:04 DEBUG http status code is 200 (expected 200) 22:30:04 DEBUG --- http response --- 22:30:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:04 DEBUG ----- output ----- 22:30:04 DEBUG { 22:30:04 DEBUG "status": "success", 22:30:04 DEBUG "data": { 22:30:04 DEBUG "resultType": "vector", 22:30:04 DEBUG "result": [ 22:30:04 DEBUG { 22:30:04 DEBUG "metric": { 22:30:04 DEBUG "instance": "10.142.0.100:9100" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG [...] 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG "instance": "10.142.0.99:9100" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 1686691804, 22:30:04 DEBUG "0.0006000000000009701" 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG } 22:30:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691804" 22:30:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691804" 22:30:04 INFO [http_cmd]: http status code OK 22:30:04 INFO [http_cmd]: http status code OK 22:30:04 DEBUG --- status code --- 22:30:04 DEBUG --- status code --- 22:30:04 DEBUG http status code is 200 (expected 200) 22:30:04 DEBUG http status code is 200 (expected 200) 22:30:04 DEBUG --- http response --- 22:30:04 DEBUG --- http response --- 22:30:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:04 DEBUG ----- output ----- 22:30:04 DEBUG ----- output ----- 22:30:04 DEBUG { 22:30:04 DEBUG { 22:30:04 DEBUG "status": "success", 22:30:04 DEBUG "status": "success", 22:30:04 DEBUG "data": { 22:30:04 DEBUG "data": { 22:30:04 DEBUG "resultType": "vector", 22:30:04 DEBUG "resultType": "vector", 22:30:04 DEBUG "result": [ 22:30:04 DEBUG "result": [ 22:30:04 DEBUG { 22:30:04 DEBUG { 22:30:04 DEBUG "metric": { 22:30:04 DEBUG "metric": { 22:30:04 DEBUG "instance": "10.142.0.100:9100" 22:30:04 DEBUG "instance": "10.142.0.100:9100" 22:30:04 DEBUG }, 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG [...] 22:30:04 DEBUG [...] 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG "instance": "10.142.0.99:9100" 22:30:04 DEBUG "instance": "10.142.0.99:9100" 22:30:04 DEBUG }, 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 1686691804, 22:30:04 DEBUG 1686691804, 22:30:04 DEBUG "20738401.033333335" 22:30:04 DEBUG "20734179.666666668" 22:30:04 DEBUG ] 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG } 22:30:04 DEBUG ] 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG } 22:30:04 DEBUG } 22:30:04 DEBUG } 22:30:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691804" 22:30:04 INFO [http_cmd]: http status code OK 22:30:04 DEBUG --- status code --- 22:30:04 DEBUG http status code is 200 (expected 200) 22:30:04 DEBUG --- http response --- 22:30:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:04 DEBUG ----- output ----- 22:30:04 DEBUG { 22:30:04 DEBUG "status": "success", 22:30:04 DEBUG "data": { 22:30:04 DEBUG "resultType": "vector", 22:30:04 DEBUG "result": [ 22:30:04 DEBUG { 22:30:04 DEBUG "metric": { 22:30:04 DEBUG "pod": "ds-idrepo-2" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG [...] 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG "pod": "ds-idrepo-0" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 1686691804, 22:30:04 DEBUG "647.3666666666667" 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG } 22:30:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691804" 22:30:04 INFO [http_cmd]: http status code OK 22:30:04 DEBUG --- status code --- 22:30:04 DEBUG http status code is 200 (expected 200) 22:30:04 DEBUG --- http response --- 22:30:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:30:04 DEBUG ----- output ----- 22:30:04 DEBUG { 22:30:04 DEBUG "status": "success", 22:30:04 DEBUG "data": { 22:30:04 DEBUG "resultType": "vector", 22:30:04 DEBUG "result": [ 22:30:04 DEBUG { 22:30:04 DEBUG "metric": { 22:30:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG [...] 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG "pod": "lodemon-6455d5ff7c-zzndq" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 1686691804, 22:30:04 DEBUG "0" 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG } 22:30:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691804" 22:30:04 INFO [http_cmd]: http status code OK 22:30:04 DEBUG --- status code --- 22:30:04 DEBUG http status code is 200 (expected 200) 22:30:04 DEBUG --- http response --- 22:30:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:04 DEBUG ----- output ----- 22:30:04 DEBUG { 22:30:04 DEBUG "status": "success", 22:30:04 DEBUG "data": { 22:30:04 DEBUG "resultType": "vector", 22:30:04 DEBUG "result": [ 22:30:04 DEBUG { 22:30:04 DEBUG "metric": { 22:30:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG [...] 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG 22:30:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:30:04 DEBUG }, 22:30:04 DEBUG "value": [ 22:30:04 DEBUG 1686691804, 22:30:04 DEBUG "0.23659720715434207" 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG ] 22:30:04 DEBUG } 22:30:04 DEBUG } 22:30:21 INFO 22:30:21 INFO [loop_until]: kubectl --namespace=xlou top pods 22:30:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:30:21 INFO [loop_until]: OK (rc = 0) 22:30:21 DEBUG --- stdout --- 22:30:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5757Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 416Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3522m 13801Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3215m 4936Mi idm-58dc667486-rgz9j 2895m 5096Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 218m 641Mi 22:30:21 DEBUG --- stderr --- 22:30:21 DEBUG 22:30:24 INFO 22:30:24 INFO [loop_until]: kubectl --namespace=xlou top node 22:30:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:30:25 INFO [loop_until]: OK (rc = 0) 22:30:25 DEBUG --- stdout --- 22:30:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3277m 20% 5878Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 69m 0% 6832Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 3077m 19% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 460m 2% 3121Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 67m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13723Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3569m 22% 14505Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 69m 0% 14601Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 281m 1% 1959Mi 3% 22:30:25 DEBUG --- stderr --- 22:30:25 DEBUG 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "ds-cts-2" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "ds-idrepo-2" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "ds-idrepo-0" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "595.0666666666666" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "ds-cts-0" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "ds-idrepo-0" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "595.0333333333333" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "ds-cts-0" 22:31:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:31:03 DEBUG }, 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "ds-idrepo-0" 22:31:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:31:03 DEBUG }, 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0" 22:31:03 DEBUG "0" 22:31:03 DEBUG ] 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:31:03 DEBUG } 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "ds-cts-2" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0.2" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "ds-cts-2" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "ds-cts-0" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "ds-cts-2" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691863" 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "instance": "10.142.0.100:9100" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "instance": "10.142.0.99:9100" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691863" 22:31:03 INFO [http_cmd]: http status code OK 22:31:03 DEBUG --- status code --- 22:31:03 DEBUG http status code is 200 (expected 200) 22:31:03 DEBUG --- http response --- 22:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:03 DEBUG ----- output ----- 22:31:03 DEBUG { 22:31:03 DEBUG "status": "success", 22:31:03 DEBUG "data": { 22:31:03 DEBUG "resultType": "vector", 22:31:03 DEBUG "result": [ 22:31:03 DEBUG { 22:31:03 DEBUG "metric": { 22:31:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG [...] 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG 22:31:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:31:03 DEBUG }, 22:31:03 DEBUG "value": [ 22:31:03 DEBUG 1686691863, 22:31:03 DEBUG "0" 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG ] 22:31:03 DEBUG } 22:31:03 DEBUG } 22:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691863" 22:31:04 INFO [http_cmd]: http status code OK 22:31:04 DEBUG --- status code --- 22:31:04 DEBUG http status code is 200 (expected 200) 22:31:04 DEBUG --- http response --- 22:31:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:04 DEBUG ----- output ----- 22:31:04 DEBUG { 22:31:04 DEBUG "status": "success", 22:31:04 DEBUG "data": { 22:31:04 DEBUG "resultType": "vector", 22:31:04 DEBUG "result": [ 22:31:04 DEBUG { 22:31:04 DEBUG "metric": { 22:31:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG [...] 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 1686691863, 22:31:04 DEBUG "13615861760" 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG } 22:31:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691864" 22:31:04 INFO [http_cmd]: http status code OK 22:31:04 DEBUG --- status code --- 22:31:04 DEBUG http status code is 200 (expected 200) 22:31:04 DEBUG --- http response --- 22:31:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:04 DEBUG ----- output ----- 22:31:04 DEBUG { 22:31:04 DEBUG "status": "success", 22:31:04 DEBUG "data": { 22:31:04 DEBUG "resultType": "vector", 22:31:04 DEBUG "result": [ 22:31:04 DEBUG { 22:31:04 DEBUG "metric": { 22:31:04 DEBUG "pod": "ds-idrepo-2" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG [...] 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG "pod": "ds-idrepo-0" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 1686691864, 22:31:04 DEBUG "595.0333333333333" 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG } 22:31:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691864" 22:31:04 INFO [http_cmd]: http status code OK 22:31:04 DEBUG --- status code --- 22:31:04 DEBUG http status code is 200 (expected 200) 22:31:04 DEBUG --- http response --- 22:31:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:04 DEBUG ----- output ----- 22:31:04 DEBUG { 22:31:04 DEBUG "status": "success", 22:31:04 DEBUG "data": { 22:31:04 DEBUG "resultType": "vector", 22:31:04 DEBUG "result": [ 22:31:04 DEBUG { 22:31:04 DEBUG "metric": { 22:31:04 DEBUG "instance": "10.142.0.100:9100" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG [...] 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG "instance": "10.142.0.99:9100" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 1686691864, 22:31:04 DEBUG "0.0003333333333330302" 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG } 22:31:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691864" 22:31:04 INFO [http_cmd]: http status code OK 22:31:04 DEBUG --- status code --- 22:31:04 DEBUG http status code is 200 (expected 200) 22:31:04 DEBUG --- http response --- 22:31:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:04 DEBUG ----- output ----- 22:31:04 DEBUG { 22:31:04 DEBUG "status": "success", 22:31:04 DEBUG "data": { 22:31:04 DEBUG "resultType": "vector", 22:31:04 DEBUG "result": [ 22:31:04 DEBUG { 22:31:04 DEBUG "metric": { 22:31:04 DEBUG "instance": "10.142.0.100:9100" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG [...] 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG "instance": "10.142.0.99:9100" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 1686691864, 22:31:04 DEBUG "19113620.9" 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG } 22:31:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691864" 22:31:04 INFO [http_cmd]: http status code OK 22:31:04 DEBUG --- status code --- 22:31:04 DEBUG http status code is 200 (expected 200) 22:31:04 DEBUG --- http response --- 22:31:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:04 DEBUG ----- output ----- 22:31:04 DEBUG { 22:31:04 DEBUG "status": "success", 22:31:04 DEBUG "data": { 22:31:04 DEBUG "resultType": "vector", 22:31:04 DEBUG "result": [ 22:31:04 DEBUG { 22:31:04 DEBUG "metric": { 22:31:04 DEBUG "instance": "10.142.0.100:9100" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG [...] 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG "instance": "10.142.0.99:9100" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 1686691864, 22:31:04 DEBUG "19113165.133333333" 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG } 22:31:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691864" 22:31:04 INFO [http_cmd]: http status code OK 22:31:04 DEBUG --- status code --- 22:31:04 DEBUG http status code is 200 (expected 200) 22:31:04 DEBUG --- http response --- 22:31:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:04 DEBUG ----- output ----- 22:31:04 DEBUG { 22:31:04 DEBUG "status": "success", 22:31:04 DEBUG "data": { 22:31:04 DEBUG "resultType": "vector", 22:31:04 DEBUG "result": [ 22:31:04 DEBUG { 22:31:04 DEBUG "metric": { 22:31:04 DEBUG "pod": "ds-idrepo-2" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG [...] 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG "pod": "ds-idrepo-0" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 1686691864, 22:31:04 DEBUG "595.0333333333333" 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG } 22:31:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691864" 22:31:04 INFO [http_cmd]: http status code OK 22:31:04 DEBUG --- status code --- 22:31:04 DEBUG http status code is 200 (expected 200) 22:31:04 DEBUG --- http response --- 22:31:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:31:04 DEBUG ----- output ----- 22:31:04 DEBUG { 22:31:04 DEBUG "status": "success", 22:31:04 DEBUG "data": { 22:31:04 DEBUG "resultType": "vector", 22:31:04 DEBUG "result": [ 22:31:04 DEBUG { 22:31:04 DEBUG "metric": { 22:31:04 DEBUG "pod": "ds-cts-2" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG [...] 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG "pod": "ds-idrepo-0" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 1686691864, 22:31:04 DEBUG "0.2855538922155688" 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG } 22:31:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691864" 22:31:04 INFO [http_cmd]: http status code OK 22:31:04 DEBUG --- status code --- 22:31:04 DEBUG http status code is 200 (expected 200) 22:31:04 DEBUG --- http response --- 22:31:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:04 DEBUG ----- output ----- 22:31:04 DEBUG { 22:31:04 DEBUG "status": "success", 22:31:04 DEBUG "data": { 22:31:04 DEBUG "resultType": "vector", 22:31:04 DEBUG "result": [ 22:31:04 DEBUG { 22:31:04 DEBUG "metric": { 22:31:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG [...] 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG 22:31:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:31:04 DEBUG }, 22:31:04 DEBUG "value": [ 22:31:04 DEBUG 1686691864, 22:31:04 DEBUG "0.22945319077344775" 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG ] 22:31:04 DEBUG } 22:31:04 DEBUG } 22:31:21 INFO 22:31:21 INFO [loop_until]: kubectl --namespace=xlou top pods 22:31:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:31:21 INFO [loop_until]: OK (rc = 0) 22:31:21 DEBUG --- stdout --- 22:31:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5757Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 7m 5688Mi ds-cts-0 10m 416Mi ds-cts-1 9m 425Mi ds-cts-2 11m 393Mi ds-idrepo-0 3149m 13802Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2812m 4937Mi idm-58dc667486-rgz9j 2717m 5096Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 206m 642Mi 22:31:21 DEBUG --- stderr --- 22:31:21 DEBUG 22:31:25 INFO 22:31:25 INFO [loop_until]: kubectl --namespace=xlou top node 22:31:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:31:25 INFO [loop_until]: OK (rc = 0) 22:31:25 DEBUG --- stdout --- 22:31:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2827m 17% 5879Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 79m 0% 6844Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2669m 16% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 441m 2% 3129Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 64m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13722Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3282m 20% 14507Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 75m 0% 14601Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 258m 1% 1958Mi 3% 22:31:25 DEBUG --- stderr --- 22:31:25 DEBUG 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG --- http response --- 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG ----- output ----- 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG { 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG { 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:03 DEBUG "data": { 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG { 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:03 DEBUG "data": { 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:03 DEBUG { 22:32:03 DEBUG "result": [ 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG { 22:32:03 DEBUG "data": { 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:03 DEBUG { 22:32:03 DEBUG "result": [ 22:32:03 DEBUG { 22:32:03 DEBUG "data": { 22:32:03 DEBUG "metric": { 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG { 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG "pod": "ds-idrepo-2" 22:32:03 DEBUG "result": [ 22:32:03 DEBUG { 22:32:03 DEBUG "data": { 22:32:03 DEBUG "metric": { 22:32:03 DEBUG "data": { 22:32:03 DEBUG "result": [ 22:32:03 DEBUG }, 22:32:03 DEBUG { 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG { 22:32:03 DEBUG "value": [ 22:32:03 DEBUG "metric": { 22:32:03 DEBUG "data": { 22:32:03 DEBUG "result": [ 22:32:03 DEBUG }, 22:32:03 DEBUG "result": [ 22:32:03 DEBUG "metric": { 22:32:03 DEBUG 22:32:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG { 22:32:03 DEBUG "value": [ 22:32:03 DEBUG { 22:32:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:32:03 DEBUG 22:32:03 DEBUG }, 22:32:03 DEBUG "result": [ 22:32:03 DEBUG "metric": { 22:32:03 DEBUG 22:32:03 DEBUG "metric": { 22:32:03 DEBUG }, 22:32:03 DEBUG 22:32:03 DEBUG "value": [ 22:32:03 DEBUG { 22:32:03 DEBUG "pod": "ds-cts-0" 22:32:03 DEBUG 22:32:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:32:03 DEBUG "value": [ 22:32:03 DEBUG [...] 22:32:03 DEBUG 22:32:03 DEBUG "metric": { 22:32:03 DEBUG }, 22:32:03 DEBUG 22:32:03 DEBUG }, 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG "pod": "ds-cts-0" 22:32:03 DEBUG "value": [ 22:32:03 DEBUG [...] 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG }, 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG [...] 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG [...] 22:32:03 DEBUG "pod": "ds-idrepo-0" 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG }, 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG [...] 22:32:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:32:03 DEBUG [...] 22:32:03 DEBUG 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG }, 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:32:03 DEBUG [...] 22:32:03 DEBUG 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:32:03 DEBUG "616.3" 22:32:03 DEBUG }, 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG 22:32:03 DEBUG }, 22:32:03 DEBUG ] 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG "pod": "ds-idrepo-0" 22:32:03 DEBUG "1.175571056772392" 22:32:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:32:03 DEBUG "value": [ 22:32:03 DEBUG } 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG 22:32:03 DEBUG }, 22:32:03 DEBUG ] 22:32:03 DEBUG }, 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG ] 22:32:03 DEBUG "0.03333333333333333" 22:32:03 DEBUG "pod": "ds-idrepo-0" 22:32:03 DEBUG "value": [ 22:32:03 DEBUG } 22:32:03 DEBUG "value": [ 22:32:03 DEBUG "0" 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG }, 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG ] 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 DEBUG "value": [ 22:32:03 DEBUG "0" 22:32:03 DEBUG } 22:32:03 DEBUG "29910.105255272127" 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG "616.3666666666667" 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG ] 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG { 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG "data": { 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG "result": [ 22:32:03 DEBUG { 22:32:03 DEBUG "metric": { 22:32:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG [...] 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG "0.03333333333333333" 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG { 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG "data": { 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG "result": [ 22:32:03 DEBUG { 22:32:03 DEBUG "metric": { 22:32:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG [...] 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG "0.2" 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG { 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG "data": { 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG "result": [ 22:32:03 DEBUG { 22:32:03 DEBUG "metric": { 22:32:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG [...] 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG "0" 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691923" 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG { 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG "data": { 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG "result": [ 22:32:03 DEBUG { 22:32:03 DEBUG "metric": { 22:32:03 DEBUG "pod": "ds-cts-0" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG [...] 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG "pod": "ds-cts-2" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG "0" 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691923" 22:32:03 INFO [http_cmd]: http status code OK 22:32:03 DEBUG --- status code --- 22:32:03 DEBUG http status code is 200 (expected 200) 22:32:03 DEBUG --- http response --- 22:32:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:03 DEBUG ----- output ----- 22:32:03 DEBUG { 22:32:03 DEBUG "status": "success", 22:32:03 DEBUG "data": { 22:32:03 DEBUG "resultType": "vector", 22:32:03 DEBUG "result": [ 22:32:03 DEBUG { 22:32:03 DEBUG "metric": { 22:32:03 DEBUG "instance": "10.142.0.100:9100" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG [...] 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG 22:32:03 DEBUG "instance": "10.142.0.99:9100" 22:32:03 DEBUG }, 22:32:03 DEBUG "value": [ 22:32:03 DEBUG 1686691923, 22:32:03 DEBUG "0" 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG ] 22:32:03 DEBUG } 22:32:03 DEBUG } 22:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691924" 22:32:04 INFO [http_cmd]: http status code OK 22:32:04 DEBUG --- status code --- 22:32:04 DEBUG http status code is 200 (expected 200) 22:32:04 DEBUG --- http response --- 22:32:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:04 DEBUG ----- output ----- 22:32:04 DEBUG { 22:32:04 DEBUG "status": "success", 22:32:04 DEBUG "data": { 22:32:04 DEBUG "resultType": "vector", 22:32:04 DEBUG "result": [ 22:32:04 DEBUG { 22:32:04 DEBUG "metric": { 22:32:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG [...] 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 1686691924, 22:32:04 DEBUG "0" 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG } 22:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691924" 22:32:04 INFO [http_cmd]: http status code OK 22:32:04 DEBUG --- status code --- 22:32:04 DEBUG http status code is 200 (expected 200) 22:32:04 DEBUG --- http response --- 22:32:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:04 DEBUG ----- output ----- 22:32:04 DEBUG { 22:32:04 DEBUG "status": "success", 22:32:04 DEBUG "data": { 22:32:04 DEBUG "resultType": "vector", 22:32:04 DEBUG "result": [ 22:32:04 DEBUG { 22:32:04 DEBUG "metric": { 22:32:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG [...] 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 1686691924, 22:32:04 DEBUG "13616496640" 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG } 22:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691924" 22:32:04 INFO [http_cmd]: http status code OK 22:32:04 DEBUG --- status code --- 22:32:04 DEBUG http status code is 200 (expected 200) 22:32:04 DEBUG --- http response --- 22:32:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:04 DEBUG ----- output ----- 22:32:04 DEBUG { 22:32:04 DEBUG "status": "success", 22:32:04 DEBUG "data": { 22:32:04 DEBUG "resultType": "vector", 22:32:04 DEBUG "result": [ 22:32:04 DEBUG { 22:32:04 DEBUG "metric": { 22:32:04 DEBUG "pod": "ds-idrepo-2" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG [...] 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG "pod": "ds-idrepo-0" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 1686691924, 22:32:04 DEBUG "616.3" 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG } 22:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691924" 22:32:04 INFO [http_cmd]: http status code OK 22:32:04 DEBUG --- status code --- 22:32:04 DEBUG http status code is 200 (expected 200) 22:32:04 DEBUG --- http response --- 22:32:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:04 DEBUG ----- output ----- 22:32:04 DEBUG { 22:32:04 DEBUG "status": "success", 22:32:04 DEBUG "data": { 22:32:04 DEBUG "resultType": "vector", 22:32:04 DEBUG "result": [ 22:32:04 DEBUG { 22:32:04 DEBUG "metric": { 22:32:04 DEBUG "instance": "10.142.0.100:9100" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG [...] 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG "instance": "10.142.0.99:9100" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 1686691924, 22:32:04 DEBUG "0.00036666666666747006" 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG } 22:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691924" 22:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691924" 22:32:04 INFO [http_cmd]: http status code OK 22:32:04 DEBUG --- status code --- 22:32:04 DEBUG http status code is 200 (expected 200) 22:32:04 DEBUG --- http response --- 22:32:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:04 DEBUG ----- output ----- 22:32:04 DEBUG { 22:32:04 DEBUG "status": "success", 22:32:04 DEBUG "data": { 22:32:04 DEBUG "resultType": "vector", 22:32:04 DEBUG "result": [ 22:32:04 DEBUG { 22:32:04 DEBUG "metric": { 22:32:04 DEBUG "instance": "10.142.0.100:9100" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG [...] 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG "instance": "10.142.0.99:9100" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 1686691924, 22:32:04 DEBUG "20680190.633333333" 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG } 22:32:04 INFO [http_cmd]: http status code OK 22:32:04 DEBUG --- status code --- 22:32:04 DEBUG http status code is 200 (expected 200) 22:32:04 DEBUG --- http response --- 22:32:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:04 DEBUG ----- output ----- 22:32:04 DEBUG { 22:32:04 DEBUG "status": "success", 22:32:04 DEBUG "data": { 22:32:04 DEBUG "resultType": "vector", 22:32:04 DEBUG "result": [ 22:32:04 DEBUG { 22:32:04 DEBUG "metric": { 22:32:04 DEBUG "instance": "10.142.0.100:9100" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG [...] 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG "instance": "10.142.0.99:9100" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 1686691924, 22:32:04 DEBUG "20676020.066666666" 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG } 22:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691924" 22:32:04 INFO [http_cmd]: http status code OK 22:32:04 DEBUG --- status code --- 22:32:04 DEBUG http status code is 200 (expected 200) 22:32:04 DEBUG --- http response --- 22:32:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:04 DEBUG ----- output ----- 22:32:04 DEBUG { 22:32:04 DEBUG "status": "success", 22:32:04 DEBUG "data": { 22:32:04 DEBUG "resultType": "vector", 22:32:04 DEBUG "result": [ 22:32:04 DEBUG { 22:32:04 DEBUG "metric": { 22:32:04 DEBUG "pod": "ds-idrepo-2" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG [...] 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG "pod": "ds-idrepo-0" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 1686691924, 22:32:04 DEBUG "616.2333333333333" 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG } 22:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691924" 22:32:04 INFO [http_cmd]: http status code OK 22:32:04 DEBUG --- status code --- 22:32:04 DEBUG http status code is 200 (expected 200) 22:32:04 DEBUG --- http response --- 22:32:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:04 DEBUG ----- output ----- 22:32:04 DEBUG { 22:32:04 DEBUG "status": "success", 22:32:04 DEBUG "data": { 22:32:04 DEBUG "resultType": "vector", 22:32:04 DEBUG "result": [ 22:32:04 DEBUG { 22:32:04 DEBUG "metric": { 22:32:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG [...] 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG "pod": "idm-58dc667486-rgz9j" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 1686691924, 22:32:04 DEBUG "0" 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG } 22:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691924" 22:32:04 INFO [http_cmd]: http status code OK 22:32:04 DEBUG --- status code --- 22:32:04 DEBUG http status code is 200 (expected 200) 22:32:04 DEBUG --- http response --- 22:32:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:04 DEBUG ----- output ----- 22:32:04 DEBUG { 22:32:04 DEBUG "status": "success", 22:32:04 DEBUG "data": { 22:32:04 DEBUG "resultType": "vector", 22:32:04 DEBUG "result": [ 22:32:04 DEBUG { 22:32:04 DEBUG "metric": { 22:32:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG [...] 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG 22:32:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:32:04 DEBUG }, 22:32:04 DEBUG "value": [ 22:32:04 DEBUG 1686691924, 22:32:04 DEBUG "0.23365877410891958" 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG ] 22:32:04 DEBUG } 22:32:04 DEBUG } 22:32:21 INFO 22:32:21 INFO [loop_until]: kubectl --namespace=xlou top pods 22:32:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:21 INFO [loop_until]: OK (rc = 0) 22:32:21 DEBUG --- stdout --- 22:32:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5757Mi am-7849cf7bdb-ntqzs 16m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 11m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3250m 13800Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3230m 4939Mi idm-58dc667486-rgz9j 2401m 5097Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 205m 642Mi 22:32:21 DEBUG --- stderr --- 22:32:21 DEBUG 22:32:25 INFO 22:32:25 INFO [loop_until]: kubectl --namespace=xlou top node 22:32:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:25 INFO [loop_until]: OK (rc = 0) 22:32:25 DEBUG --- stdout --- 22:32:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3376m 21% 5880Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 80m 0% 6829Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2364m 14% 6075Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 445m 2% 3132Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 86m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1173Mi 2% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1200Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13722Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3296m 20% 14536Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 66m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 69m 0% 14600Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 270m 1% 1959Mi 3% 22:32:25 DEBUG --- stderr --- 22:32:25 DEBUG 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG { 22:33:03 DEBUG { 22:33:03 DEBUG { 22:33:03 DEBUG { 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "data": { 22:33:03 DEBUG "data": { 22:33:03 DEBUG "data": { 22:33:03 DEBUG "data": { 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG "result": [ 22:33:03 DEBUG "result": [ 22:33:03 DEBUG "result": [ 22:33:03 DEBUG "result": [ 22:33:03 DEBUG { 22:33:03 DEBUG { 22:33:03 DEBUG { 22:33:03 DEBUG { 22:33:03 DEBUG "metric": { 22:33:03 DEBUG "metric": { 22:33:03 DEBUG "metric": { 22:33:03 DEBUG "metric": { 22:33:03 DEBUG "pod": "ds-cts-0" 22:33:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:33:03 DEBUG "pod": "ds-idrepo-2" 22:33:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:33:03 DEBUG }, 22:33:03 DEBUG }, 22:33:03 DEBUG }, 22:33:03 DEBUG }, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG "value": [ 22:33:03 DEBUG "value": [ 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG [...] 22:33:03 DEBUG [...] 22:33:03 DEBUG [...] 22:33:03 DEBUG [...] 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 DEBUG "pod": "ds-idrepo-0" 22:33:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:33:03 DEBUG "pod": "ds-idrepo-0" 22:33:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG }, 22:33:03 DEBUG }, 22:33:03 DEBUG }, 22:33:03 DEBUG }, 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG "value": [ 22:33:03 DEBUG "value": [ 22:33:03 DEBUG "value": [ 22:33:03 DEBUG "value": [ 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG "576.5" 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 DEBUG "0" 22:33:03 DEBUG "576.5" 22:33:03 DEBUG "0" 22:33:03 DEBUG ] 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG ] 22:33:03 DEBUG ] 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG { 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 DEBUG ] 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG ] 22:33:03 DEBUG ] 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG "data": { 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 DEBUG "result": [ 22:33:03 DEBUG { 22:33:03 DEBUG { 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "metric": { 22:33:03 DEBUG "data": { 22:33:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG }, 22:33:03 DEBUG "result": [ 22:33:03 DEBUG "value": [ 22:33:03 DEBUG { 22:33:03 DEBUG 22:33:03 DEBUG "metric": { 22:33:03 DEBUG 22:33:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:33:03 DEBUG 22:33:03 DEBUG }, 22:33:03 DEBUG [...] 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:33:03 DEBUG [...] 22:33:03 DEBUG }, 22:33:03 DEBUG 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG 22:33:03 DEBUG "0" 22:33:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:33:03 DEBUG ] 22:33:03 DEBUG }, 22:33:03 DEBUG } 22:33:03 DEBUG "value": [ 22:33:03 DEBUG ] 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG } 22:33:03 DEBUG "0.00775137200349399" 22:33:03 DEBUG } 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 DEBUG --- status code --- 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:03 DEBUG { 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG { 22:33:03 DEBUG "data": { 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG "data": { 22:33:03 DEBUG "result": [ 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG { 22:33:03 DEBUG "result": [ 22:33:03 DEBUG "metric": { 22:33:03 DEBUG { 22:33:03 DEBUG "pod": "ds-cts-0" 22:33:03 DEBUG "metric": { 22:33:03 DEBUG }, 22:33:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:33:03 DEBUG "value": [ 22:33:03 DEBUG }, 22:33:03 DEBUG 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG [...] 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG [...] 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG "pod": "ds-idrepo-0" 22:33:03 DEBUG 22:33:03 DEBUG }, 22:33:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:33:03 DEBUG "value": [ 22:33:03 DEBUG }, 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG "0" 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG ] 22:33:03 DEBUG "193816290.64087218" 22:33:03 DEBUG } 22:33:03 DEBUG ] 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG { 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "data": { 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG "result": [ 22:33:03 DEBUG { 22:33:03 DEBUG "metric": { 22:33:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:33:03 DEBUG }, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG [...] 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:33:03 DEBUG }, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG "0" 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG { 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "data": { 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG "result": [ 22:33:03 DEBUG { 22:33:03 DEBUG "metric": { 22:33:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:33:03 DEBUG }, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG [...] 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:33:03 DEBUG }, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG "0.2" 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691983" 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG { 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "data": { 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG "result": [ 22:33:03 DEBUG { 22:33:03 DEBUG "metric": { 22:33:03 DEBUG "pod": "ds-cts-0" 22:33:03 DEBUG }, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG [...] 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG "pod": "ds-cts-2" 22:33:03 DEBUG }, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG "0" 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691983" 22:33:03 INFO [http_cmd]: http status code OK 22:33:03 DEBUG --- status code --- 22:33:03 DEBUG http status code is 200 (expected 200) 22:33:03 DEBUG --- http response --- 22:33:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:03 DEBUG ----- output ----- 22:33:03 DEBUG { 22:33:03 DEBUG "status": "success", 22:33:03 DEBUG "data": { 22:33:03 DEBUG "resultType": "vector", 22:33:03 DEBUG "result": [ 22:33:03 DEBUG { 22:33:03 DEBUG "metric": { 22:33:03 DEBUG "instance": "10.142.0.100:9100" 22:33:03 DEBUG }, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG [...] 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG 22:33:03 DEBUG "instance": "10.142.0.99:9100" 22:33:03 DEBUG }, 22:33:03 DEBUG "value": [ 22:33:03 DEBUG 1686691983, 22:33:03 DEBUG "0" 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG ] 22:33:03 DEBUG } 22:33:03 DEBUG } 22:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691984" 22:33:04 INFO [http_cmd]: http status code OK 22:33:04 DEBUG --- status code --- 22:33:04 DEBUG http status code is 200 (expected 200) 22:33:04 DEBUG --- http response --- 22:33:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:04 DEBUG ----- output ----- 22:33:04 DEBUG { 22:33:04 DEBUG "status": "success", 22:33:04 DEBUG "data": { 22:33:04 DEBUG "resultType": "vector", 22:33:04 DEBUG "result": [ 22:33:04 DEBUG { 22:33:04 DEBUG "metric": { 22:33:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG [...] 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 1686691984, 22:33:04 DEBUG "0" 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG } 22:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686691984" 22:33:04 INFO [http_cmd]: http status code OK 22:33:04 DEBUG --- status code --- 22:33:04 DEBUG http status code is 200 (expected 200) 22:33:04 DEBUG --- http response --- 22:33:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:04 DEBUG ----- output ----- 22:33:04 DEBUG { 22:33:04 DEBUG "status": "success", 22:33:04 DEBUG "data": { 22:33:04 DEBUG "resultType": "vector", 22:33:04 DEBUG "result": [ 22:33:04 DEBUG { 22:33:04 DEBUG "metric": { 22:33:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG [...] 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 1686691984, 22:33:04 DEBUG "13617782784" 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG } 22:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691984" 22:33:04 INFO [http_cmd]: http status code OK 22:33:04 DEBUG --- status code --- 22:33:04 DEBUG http status code is 200 (expected 200) 22:33:04 DEBUG --- http response --- 22:33:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:04 DEBUG ----- output ----- 22:33:04 DEBUG { 22:33:04 DEBUG "status": "success", 22:33:04 DEBUG "data": { 22:33:04 DEBUG "resultType": "vector", 22:33:04 DEBUG "result": [ 22:33:04 DEBUG { 22:33:04 DEBUG "metric": { 22:33:04 DEBUG "pod": "ds-idrepo-2" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG [...] 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG "pod": "ds-idrepo-0" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 1686691984, 22:33:04 DEBUG "576.5" 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG } 22:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691984" 22:33:04 INFO [http_cmd]: http status code OK 22:33:04 DEBUG --- status code --- 22:33:04 DEBUG http status code is 200 (expected 200) 22:33:04 DEBUG --- http response --- 22:33:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:04 DEBUG ----- output ----- 22:33:04 DEBUG { 22:33:04 DEBUG "status": "success", 22:33:04 DEBUG "data": { 22:33:04 DEBUG "resultType": "vector", 22:33:04 DEBUG "result": [ 22:33:04 DEBUG { 22:33:04 DEBUG "metric": { 22:33:04 DEBUG "instance": "10.142.0.100:9100" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG [...] 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG "instance": "10.142.0.99:9100" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 1686691984, 22:33:04 DEBUG "0.00046666666666700016" 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG } 22:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691984" 22:33:04 INFO [http_cmd]: http status code OK 22:33:04 DEBUG --- status code --- 22:33:04 DEBUG http status code is 200 (expected 200) 22:33:04 DEBUG --- http response --- 22:33:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:04 DEBUG ----- output ----- 22:33:04 DEBUG { 22:33:04 DEBUG "status": "success", 22:33:04 DEBUG "data": { 22:33:04 DEBUG "resultType": "vector", 22:33:04 DEBUG "result": [ 22:33:04 DEBUG { 22:33:04 DEBUG "metric": { 22:33:04 DEBUG "instance": "10.142.0.100:9100" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG [...] 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG "instance": "10.142.0.99:9100" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 1686691984, 22:33:04 DEBUG "20688728.233333334" 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG } 22:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686691984" 22:33:04 INFO [http_cmd]: http status code OK 22:33:04 DEBUG --- status code --- 22:33:04 DEBUG http status code is 200 (expected 200) 22:33:04 DEBUG --- http response --- 22:33:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:04 DEBUG ----- output ----- 22:33:04 DEBUG { 22:33:04 DEBUG "status": "success", 22:33:04 DEBUG "data": { 22:33:04 DEBUG "resultType": "vector", 22:33:04 DEBUG "result": [ 22:33:04 DEBUG { 22:33:04 DEBUG "metric": { 22:33:04 DEBUG "instance": "10.142.0.100:9100" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG [...] 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG "instance": "10.142.0.99:9100" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 1686691984, 22:33:04 DEBUG "20684422.066666666" 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG } 22:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691984" 22:33:04 INFO [http_cmd]: http status code OK 22:33:04 DEBUG --- status code --- 22:33:04 DEBUG http status code is 200 (expected 200) 22:33:04 DEBUG --- http response --- 22:33:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:04 DEBUG ----- output ----- 22:33:04 DEBUG { 22:33:04 DEBUG "status": "success", 22:33:04 DEBUG "data": { 22:33:04 DEBUG "resultType": "vector", 22:33:04 DEBUG "result": [ 22:33:04 DEBUG { 22:33:04 DEBUG "metric": { 22:33:04 DEBUG "pod": "ds-idrepo-2" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG [...] 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG "pod": "ds-idrepo-0" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 1686691984, 22:33:04 DEBUG "576.5" 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG } 22:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686691984" 22:33:04 INFO [http_cmd]: http status code OK 22:33:04 DEBUG --- status code --- 22:33:04 DEBUG http status code is 200 (expected 200) 22:33:04 DEBUG --- http response --- 22:33:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:33:04 DEBUG ----- output ----- 22:33:04 DEBUG { 22:33:04 DEBUG "status": "success", 22:33:04 DEBUG "data": { 22:33:04 DEBUG "resultType": "vector", 22:33:04 DEBUG "result": [ 22:33:04 DEBUG { 22:33:04 DEBUG "metric": { 22:33:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG [...] 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG "pod": "ds-cts-0" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 1686691984, 22:33:04 DEBUG "0" 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG } 22:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686691984" 22:33:04 INFO [http_cmd]: http status code OK 22:33:04 DEBUG --- status code --- 22:33:04 DEBUG http status code is 200 (expected 200) 22:33:04 DEBUG --- http response --- 22:33:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:04 DEBUG ----- output ----- 22:33:04 DEBUG { 22:33:04 DEBUG "status": "success", 22:33:04 DEBUG "data": { 22:33:04 DEBUG "resultType": "vector", 22:33:04 DEBUG "result": [ 22:33:04 DEBUG { 22:33:04 DEBUG "metric": { 22:33:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG [...] 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG 22:33:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:33:04 DEBUG }, 22:33:04 DEBUG "value": [ 22:33:04 DEBUG 1686691984, 22:33:04 DEBUG "0.2274584354763843" 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG ] 22:33:04 DEBUG } 22:33:04 DEBUG } 22:33:21 INFO 22:33:21 INFO [loop_until]: kubectl --namespace=xlou top pods 22:33:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:33:21 INFO [loop_until]: OK (rc = 0) 22:33:21 DEBUG --- stdout --- 22:33:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 10m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3285m 13806Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3198m 4941Mi idm-58dc667486-rgz9j 2456m 5096Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 206m 642Mi 22:33:21 DEBUG --- stderr --- 22:33:21 DEBUG 22:33:25 INFO 22:33:25 INFO [loop_until]: kubectl --namespace=xlou top node 22:33:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:33:25 INFO [loop_until]: OK (rc = 0) 22:33:25 DEBUG --- stdout --- 22:33:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3332m 20% 5884Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6863Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 71m 0% 6831Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2780m 17% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 447m 2% 3131Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 86m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13722Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3422m 21% 14511Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14600Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 279m 1% 1961Mi 3% 22:33:25 DEBUG --- stderr --- 22:33:25 DEBUG 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG { 22:34:03 DEBUG { 22:34:03 DEBUG { 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "data": { 22:34:03 DEBUG "data": { 22:34:03 DEBUG "data": { 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "result": [ 22:34:03 DEBUG "result": [ 22:34:03 DEBUG "result": [ 22:34:03 DEBUG { 22:34:03 DEBUG { 22:34:03 DEBUG { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "pod": "ds-idrepo-2" 22:34:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:34:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:34:03 DEBUG }, 22:34:03 DEBUG }, 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG "value": [ 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG [...] 22:34:03 DEBUG [...] 22:34:03 DEBUG [...] 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG "pod": "ds-idrepo-0" 22:34:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:34:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:34:03 DEBUG }, 22:34:03 DEBUG }, 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG "value": [ 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG "640.7666666666667" 22:34:03 DEBUG "0" 22:34:03 DEBUG "0" 22:34:03 DEBUG ] 22:34:03 DEBUG ] 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG ] 22:34:03 DEBUG ] 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG { 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "data": { 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "result": [ 22:34:03 DEBUG { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "pod": "ds-cts-0" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG [...] 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG "pod": "ds-idrepo-0" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG "640.7666666666667" 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG { 22:34:03 DEBUG { 22:34:03 DEBUG { 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "data": { 22:34:03 DEBUG "data": { 22:34:03 DEBUG "data": { 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "result": [ 22:34:03 DEBUG "result": [ 22:34:03 DEBUG "result": [ 22:34:03 DEBUG { 22:34:03 DEBUG { 22:34:03 DEBUG { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:34:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:34:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:34:03 DEBUG }, 22:34:03 DEBUG }, 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG "value": [ 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG [...] 22:34:03 DEBUG [...] 22:34:03 DEBUG [...] 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:34:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:34:03 DEBUG }, 22:34:03 DEBUG }, 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG "value": [ 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG "0" 22:34:03 DEBUG "1.8809647891161596" 22:34:03 DEBUG "0" 22:34:03 DEBUG ] 22:34:03 DEBUG ] 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG ] 22:34:03 DEBUG ] 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG { 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "data": { 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "result": [ 22:34:03 DEBUG { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "pod": "ds-cts-0" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG [...] 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG "pod": "ds-idrepo-0" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG "0" 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG { 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "data": { 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "result": [ 22:34:03 DEBUG { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG [...] 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG "pod": "idm-58dc667486-rgz9j" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG "19677.004370160237" 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG { 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "data": { 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "result": [ 22:34:03 DEBUG { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG [...] 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG "0.2" 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692043" 22:34:03 INFO [http_cmd]: http status code OK 22:34:03 DEBUG --- status code --- 22:34:03 DEBUG http status code is 200 (expected 200) 22:34:03 DEBUG --- http response --- 22:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:34:03 DEBUG ----- output ----- 22:34:03 DEBUG { 22:34:03 DEBUG "status": "success", 22:34:03 DEBUG "data": { 22:34:03 DEBUG "resultType": "vector", 22:34:03 DEBUG "result": [ 22:34:03 DEBUG { 22:34:03 DEBUG "metric": { 22:34:03 DEBUG "pod": "ds-cts-0" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG [...] 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG 22:34:03 DEBUG "pod": "ds-cts-2" 22:34:03 DEBUG }, 22:34:03 DEBUG "value": [ 22:34:03 DEBUG 1686692043, 22:34:03 DEBUG "0" 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG ] 22:34:03 DEBUG } 22:34:03 DEBUG } 22:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692044" 22:34:04 INFO [http_cmd]: http status code OK 22:34:04 DEBUG --- status code --- 22:34:04 DEBUG http status code is 200 (expected 200) 22:34:04 DEBUG --- http response --- 22:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:04 DEBUG ----- output ----- 22:34:04 DEBUG { 22:34:04 DEBUG "status": "success", 22:34:04 DEBUG "data": { 22:34:04 DEBUG "resultType": "vector", 22:34:04 DEBUG "result": [ 22:34:04 DEBUG { 22:34:04 DEBUG "metric": { 22:34:04 DEBUG "instance": "10.142.0.100:9100" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG [...] 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG "instance": "10.142.0.99:9100" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 1686692044, 22:34:04 DEBUG "0" 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG } 22:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692044" 22:34:04 INFO [http_cmd]: http status code OK 22:34:04 DEBUG --- status code --- 22:34:04 DEBUG http status code is 200 (expected 200) 22:34:04 DEBUG --- http response --- 22:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:04 DEBUG ----- output ----- 22:34:04 DEBUG { 22:34:04 DEBUG "status": "success", 22:34:04 DEBUG "data": { 22:34:04 DEBUG "resultType": "vector", 22:34:04 DEBUG "result": [ 22:34:04 DEBUG { 22:34:04 DEBUG "metric": { 22:34:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG [...] 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 1686692044, 22:34:04 DEBUG "0" 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG } 22:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692044" 22:34:04 INFO [http_cmd]: http status code OK 22:34:04 DEBUG --- status code --- 22:34:04 DEBUG http status code is 200 (expected 200) 22:34:04 DEBUG --- http response --- 22:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:04 DEBUG ----- output ----- 22:34:04 DEBUG { 22:34:04 DEBUG "status": "success", 22:34:04 DEBUG "data": { 22:34:04 DEBUG "resultType": "vector", 22:34:04 DEBUG "result": [ 22:34:04 DEBUG { 22:34:04 DEBUG "metric": { 22:34:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG [...] 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 1686692044, 22:34:04 DEBUG "13617885184" 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG } 22:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692044" 22:34:04 INFO [http_cmd]: http status code OK 22:34:04 DEBUG --- status code --- 22:34:04 DEBUG http status code is 200 (expected 200) 22:34:04 DEBUG --- http response --- 22:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:04 DEBUG ----- output ----- 22:34:04 DEBUG { 22:34:04 DEBUG "status": "success", 22:34:04 DEBUG "data": { 22:34:04 DEBUG "resultType": "vector", 22:34:04 DEBUG "result": [ 22:34:04 DEBUG { 22:34:04 DEBUG "metric": { 22:34:04 DEBUG "pod": "ds-idrepo-2" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG [...] 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG "pod": "ds-idrepo-0" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 1686692044, 22:34:04 DEBUG "640.7333333333333" 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG } 22:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692044" 22:34:04 INFO [http_cmd]: http status code OK 22:34:04 DEBUG --- status code --- 22:34:04 DEBUG http status code is 200 (expected 200) 22:34:04 DEBUG --- http response --- 22:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:04 DEBUG ----- output ----- 22:34:04 DEBUG { 22:34:04 DEBUG "status": "success", 22:34:04 DEBUG "data": { 22:34:04 DEBUG "resultType": "vector", 22:34:04 DEBUG "result": [ 22:34:04 DEBUG { 22:34:04 DEBUG "metric": { 22:34:04 DEBUG "instance": "10.142.0.100:9100" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG [...] 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG "instance": "10.142.0.99:9100" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 1686692044, 22:34:04 DEBUG "0.00036666666666557526" 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG } 22:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692044" 22:34:04 INFO [http_cmd]: http status code OK 22:34:04 DEBUG --- status code --- 22:34:04 DEBUG http status code is 200 (expected 200) 22:34:04 DEBUG --- http response --- 22:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:04 DEBUG ----- output ----- 22:34:04 DEBUG { 22:34:04 DEBUG "status": "success", 22:34:04 DEBUG "data": { 22:34:04 DEBUG "resultType": "vector", 22:34:04 DEBUG "result": [ 22:34:04 DEBUG { 22:34:04 DEBUG "metric": { 22:34:04 DEBUG "instance": "10.142.0.100:9100" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG [...] 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG "instance": "10.142.0.99:9100" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 1686692044, 22:34:04 DEBUG "19796459.433333334" 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG } 22:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692044" 22:34:04 INFO [http_cmd]: http status code OK 22:34:04 DEBUG --- status code --- 22:34:04 DEBUG http status code is 200 (expected 200) 22:34:04 DEBUG --- http response --- 22:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:04 DEBUG ----- output ----- 22:34:04 DEBUG { 22:34:04 DEBUG "status": "success", 22:34:04 DEBUG "data": { 22:34:04 DEBUG "resultType": "vector", 22:34:04 DEBUG "result": [ 22:34:04 DEBUG { 22:34:04 DEBUG "metric": { 22:34:04 DEBUG "instance": "10.142.0.100:9100" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG [...] 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG "instance": "10.142.0.99:9100" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 1686692044, 22:34:04 DEBUG "19792784.599999998" 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG } 22:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692044" 22:34:04 INFO [http_cmd]: http status code OK 22:34:04 DEBUG --- status code --- 22:34:04 DEBUG http status code is 200 (expected 200) 22:34:04 DEBUG --- http response --- 22:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:04 DEBUG ----- output ----- 22:34:04 DEBUG { 22:34:04 DEBUG "status": "success", 22:34:04 DEBUG "data": { 22:34:04 DEBUG "resultType": "vector", 22:34:04 DEBUG "result": [ 22:34:04 DEBUG { 22:34:04 DEBUG "metric": { 22:34:04 DEBUG "pod": "ds-idrepo-2" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG [...] 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG "pod": "ds-idrepo-0" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 1686692044, 22:34:04 DEBUG "640.7" 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG } 22:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692044" 22:34:04 INFO [http_cmd]: http status code OK 22:34:04 DEBUG --- status code --- 22:34:04 DEBUG http status code is 200 (expected 200) 22:34:04 DEBUG --- http response --- 22:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:04 DEBUG ----- output ----- 22:34:04 DEBUG { 22:34:04 DEBUG "status": "success", 22:34:04 DEBUG "data": { 22:34:04 DEBUG "resultType": "vector", 22:34:04 DEBUG "result": [ 22:34:04 DEBUG { 22:34:04 DEBUG "metric": { 22:34:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG [...] 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG 22:34:04 DEBUG "pod": "idm-58dc667486-rgz9j" 22:34:04 DEBUG }, 22:34:04 DEBUG "value": [ 22:34:04 DEBUG 1686692044, 22:34:04 DEBUG "0" 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG ] 22:34:04 DEBUG } 22:34:04 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692045" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected 200) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1686692045, 22:34:05 DEBUG "0.22965624729286063" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:21 INFO 22:34:21 INFO [loop_until]: kubectl --namespace=xlou top pods 22:34:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:34:21 INFO [loop_until]: OK (rc = 0) 22:34:21 DEBUG --- stdout --- 22:34:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 9m 392Mi ds-idrepo-0 3349m 13829Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3099m 4943Mi idm-58dc667486-rgz9j 2752m 5097Mi lodemon-6455d5ff7c-zzndq 5m 71Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 219m 642Mi 22:34:21 DEBUG --- stderr --- 22:34:21 DEBUG 22:34:25 INFO 22:34:25 INFO [loop_until]: kubectl --namespace=xlou top node 22:34:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:34:25 INFO [loop_until]: OK (rc = 0) 22:34:25 DEBUG --- stdout --- 22:34:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3121m 19% 5884Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6825Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2801m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 438m 2% 3137Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 87m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13724Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3476m 21% 14506Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14602Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 284m 1% 1963Mi 3% 22:34:25 DEBUG --- stderr --- 22:34:25 DEBUG 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:03 INFO [http_cmd]: http status code OK 22:35:03 INFO [http_cmd]: http status code OK 22:35:03 INFO [http_cmd]: http status code OK 22:35:03 INFO [http_cmd]: http status code OK 22:35:03 DEBUG --- status code --- 22:35:03 DEBUG --- status code --- 22:35:03 DEBUG --- status code --- 22:35:03 DEBUG --- status code --- 22:35:03 DEBUG http status code is 200 (expected 200) 22:35:03 DEBUG http status code is 200 (expected 200) 22:35:03 DEBUG http status code is 200 (expected 200) 22:35:03 DEBUG http status code is 200 (expected 200) 22:35:03 DEBUG --- http response --- 22:35:03 DEBUG --- http response --- 22:35:03 DEBUG --- http response --- 22:35:03 DEBUG --- http response --- 22:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:03 INFO [http_cmd]: http status code OK 22:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:03 INFO [http_cmd]: http status code OK 22:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:03 DEBUG ----- output ----- 22:35:03 DEBUG --- status code --- 22:35:03 DEBUG ----- output ----- 22:35:03 DEBUG --- status code --- 22:35:03 DEBUG ----- output ----- 22:35:03 DEBUG ----- output ----- 22:35:03 DEBUG { 22:35:03 DEBUG http status code is 200 (expected 200) 22:35:03 DEBUG { 22:35:03 DEBUG http status code is 200 (expected 200) 22:35:03 DEBUG { 22:35:03 DEBUG { 22:35:03 DEBUG "status": "success", 22:35:03 DEBUG --- http response --- 22:35:03 DEBUG "status": "success", 22:35:03 DEBUG --- http response --- 22:35:03 DEBUG "status": "success", 22:35:03 DEBUG "status": "success", 22:35:03 DEBUG "data": { 22:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:03 DEBUG "data": { 22:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:03 DEBUG "data": { 22:35:03 DEBUG "data": { 22:35:03 DEBUG "resultType": "vector", 22:35:03 DEBUG ----- output ----- 22:35:03 DEBUG "resultType": "vector", 22:35:03 DEBUG ----- output ----- 22:35:03 DEBUG "resultType": "vector", 22:35:03 DEBUG "resultType": "vector", 22:35:03 DEBUG "result": [ 22:35:03 DEBUG { 22:35:03 DEBUG "result": [ 22:35:03 DEBUG { 22:35:03 DEBUG "result": [ 22:35:03 DEBUG "result": [ 22:35:03 DEBUG { 22:35:03 DEBUG "status": "success", 22:35:03 DEBUG { 22:35:03 DEBUG "status": "success", 22:35:03 DEBUG { 22:35:03 DEBUG { 22:35:03 DEBUG "metric": { 22:35:03 DEBUG "data": { 22:35:03 DEBUG "metric": { 22:35:03 DEBUG "data": { 22:35:03 DEBUG "metric": { 22:35:03 DEBUG "metric": { 22:35:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:35:03 DEBUG "resultType": "vector", 22:35:03 DEBUG "pod": "ds-cts-0" 22:35:03 DEBUG "resultType": "vector", 22:35:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:35:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:35:03 DEBUG }, 22:35:03 DEBUG "result": [ 22:35:03 DEBUG }, 22:35:03 DEBUG "result": [ 22:35:03 DEBUG }, 22:35:03 DEBUG }, 22:35:03 DEBUG "value": [ 22:35:03 DEBUG { 22:35:03 DEBUG "value": [ 22:35:03 DEBUG { 22:35:03 DEBUG "value": [ 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 22:35:03 DEBUG "metric": { 22:35:03 DEBUG 22:35:03 DEBUG "metric": { 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG "pod": "ds-idrepo-2" 22:35:03 DEBUG 22:35:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG }, 22:35:03 DEBUG 22:35:03 DEBUG }, 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG [...] 22:35:03 DEBUG "value": [ 22:35:03 DEBUG [...] 22:35:03 DEBUG "value": [ 22:35:03 DEBUG [...] 22:35:03 DEBUG [...] 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:35:03 DEBUG [...] 22:35:03 DEBUG "pod": "ds-idrepo-0" 22:35:03 DEBUG [...] 22:35:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:35:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:35:03 DEBUG }, 22:35:03 DEBUG 22:35:03 DEBUG }, 22:35:03 DEBUG 22:35:03 DEBUG }, 22:35:03 DEBUG }, 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 22:35:03 DEBUG "value": [ 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 1686692103, 22:35:03 DEBUG 22:35:03 DEBUG 1686692103, 22:35:03 DEBUG 22:35:03 DEBUG 1686692103, 22:35:03 DEBUG 1686692103, 22:35:03 DEBUG "0" 22:35:03 DEBUG "pod": "ds-idrepo-0" 22:35:03 DEBUG "635.6333333333333" 22:35:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:35:03 DEBUG "0" 22:35:03 DEBUG "0" 22:35:03 DEBUG ] 22:35:03 DEBUG }, 22:35:03 DEBUG ] 22:35:03 DEBUG }, 22:35:03 DEBUG ] 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG "value": [ 22:35:03 DEBUG } 22:35:03 DEBUG "value": [ 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 DEBUG ] 22:35:03 DEBUG 1686692103, 22:35:03 DEBUG ] 22:35:03 DEBUG 1686692103, 22:35:03 DEBUG ] 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG "635.6666666666666" 22:35:03 DEBUG } 22:35:03 DEBUG "0.00815476320096821" 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 DEBUG ] 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:03 INFO [http_cmd]: http status code OK 22:35:03 DEBUG --- status code --- 22:35:03 DEBUG http status code is 200 (expected 200) 22:35:03 DEBUG --- http response --- 22:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:35:03 DEBUG ----- output ----- 22:35:03 DEBUG { 22:35:03 DEBUG "status": "success", 22:35:03 DEBUG "data": { 22:35:03 DEBUG "resultType": "vector", 22:35:03 DEBUG "result": [ 22:35:03 DEBUG { 22:35:03 DEBUG "metric": { 22:35:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:35:03 DEBUG }, 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG [...] 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:35:03 DEBUG }, 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 1686692103, 22:35:03 DEBUG "0" 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 INFO [http_cmd]: http status code OK 22:35:03 DEBUG --- status code --- 22:35:03 DEBUG http status code is 200 (expected 200) 22:35:03 DEBUG --- http response --- 22:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:03 DEBUG ----- output ----- 22:35:03 DEBUG { 22:35:03 DEBUG "status": "success", 22:35:03 DEBUG "data": { 22:35:03 DEBUG "resultType": "vector", 22:35:03 DEBUG "result": [ 22:35:03 DEBUG { 22:35:03 DEBUG "metric": { 22:35:03 DEBUG "pod": "ds-cts-0" 22:35:03 DEBUG }, 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG [...] 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG "pod": "ds-idrepo-0" 22:35:03 DEBUG }, 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 1686692103, 22:35:03 DEBUG "0" 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:03 INFO [http_cmd]: http status code OK 22:35:03 DEBUG --- status code --- 22:35:03 DEBUG http status code is 200 (expected 200) 22:35:03 DEBUG --- http response --- 22:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:03 DEBUG ----- output ----- 22:35:03 DEBUG { 22:35:03 DEBUG "status": "success", 22:35:03 DEBUG "data": { 22:35:03 DEBUG "resultType": "vector", 22:35:03 DEBUG "result": [ 22:35:03 DEBUG { 22:35:03 DEBUG "metric": { 22:35:03 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:35:03 DEBUG }, 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG [...] 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG 22:35:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:35:03 DEBUG }, 22:35:03 DEBUG "value": [ 22:35:03 DEBUG 1686692103, 22:35:03 DEBUG "0" 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG ] 22:35:03 DEBUG } 22:35:03 DEBUG } 22:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692103" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692103, 22:35:04 DEBUG "0.2" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "pod": "ds-cts-0" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "pod": "ds-cts-2" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "0" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "instance": "10.142.0.100:9100" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "instance": "10.142.0.99:9100" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "0" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "0" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "13616529408" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "pod": "ds-idrepo-2" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "pod": "ds-idrepo-0" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "635.7" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "instance": "10.142.0.100:9100" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "instance": "10.142.0.99:9100" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "0.0002999999999985903" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "instance": "10.142.0.100:9100" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "instance": "10.142.0.99:9100" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "19818438.933333334" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "instance": "10.142.0.100:9100" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "instance": "10.142.0.99:9100" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "19814247.9" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "pod": "ds-idrepo-2" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "pod": "ds-idrepo-0" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "635.7" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692104" 22:35:04 INFO [http_cmd]: http status code OK 22:35:04 DEBUG --- status code --- 22:35:04 DEBUG http status code is 200 (expected 200) 22:35:04 DEBUG --- http response --- 22:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:35:04 DEBUG ----- output ----- 22:35:04 DEBUG { 22:35:04 DEBUG "status": "success", 22:35:04 DEBUG "data": { 22:35:04 DEBUG "resultType": "vector", 22:35:04 DEBUG "result": [ 22:35:04 DEBUG { 22:35:04 DEBUG "metric": { 22:35:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG [...] 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG 22:35:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:35:04 DEBUG }, 22:35:04 DEBUG "value": [ 22:35:04 DEBUG 1686692104, 22:35:04 DEBUG "0" 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG ] 22:35:04 DEBUG } 22:35:04 DEBUG } 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692105" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected 200) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1686692105, 22:35:05 DEBUG "0.23185644059384317" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:21 INFO 22:35:21 INFO [loop_until]: kubectl --namespace=xlou top pods 22:35:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:35:21 INFO [loop_until]: OK (rc = 0) 22:35:21 DEBUG --- stdout --- 22:35:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 8m 417Mi ds-cts-1 8m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3473m 13800Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3174m 4945Mi idm-58dc667486-rgz9j 2847m 5097Mi lodemon-6455d5ff7c-zzndq 5m 71Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 214m 642Mi 22:35:21 DEBUG --- stderr --- 22:35:21 DEBUG 22:35:25 INFO 22:35:25 INFO [loop_until]: kubectl --namespace=xlou top node 22:35:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:35:25 INFO [loop_until]: OK (rc = 0) 22:35:25 DEBUG --- stdout --- 22:35:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3356m 21% 5884Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6861Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6827Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2866m 18% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 455m 2% 3138Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 64m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1206Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13723Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3596m 22% 14524Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14602Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 287m 1% 1960Mi 3% 22:35:25 DEBUG --- stderr --- 22:35:25 DEBUG 22:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692163" 22:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692163" 22:36:03 INFO [http_cmd]: http status code OK 22:36:03 DEBUG --- status code --- 22:36:03 DEBUG http status code is 200 (expected 200) 22:36:03 DEBUG --- http response --- 22:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:03 INFO [http_cmd]: http status code OK 22:36:03 DEBUG ----- output ----- 22:36:03 DEBUG --- status code --- 22:36:03 DEBUG { 22:36:03 DEBUG http status code is 200 (expected 200) 22:36:03 DEBUG "status": "success", 22:36:03 DEBUG --- http response --- 22:36:03 DEBUG "data": { 22:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:03 DEBUG "resultType": "vector", 22:36:03 DEBUG ----- output ----- 22:36:03 DEBUG "result": [ 22:36:03 DEBUG { 22:36:03 DEBUG { 22:36:03 DEBUG "status": "success", 22:36:03 DEBUG "metric": { 22:36:03 DEBUG "data": { 22:36:03 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:36:03 DEBUG "resultType": "vector", 22:36:03 DEBUG }, 22:36:03 DEBUG "result": [ 22:36:03 DEBUG "value": [ 22:36:03 DEBUG { 22:36:03 DEBUG 22:36:03 DEBUG "metric": { 22:36:03 DEBUG 22:36:03 DEBUG "pod": "ds-idrepo-2" 22:36:03 DEBUG 22:36:03 DEBUG }, 22:36:03 DEBUG [...] 22:36:03 DEBUG "value": [ 22:36:03 DEBUG 22:36:03 DEBUG 22:36:03 DEBUG 22:36:03 DEBUG 22:36:03 DEBUG 22:36:03 DEBUG 22:36:03 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:36:03 DEBUG [...] 22:36:03 DEBUG }, 22:36:03 DEBUG 22:36:03 DEBUG "value": [ 22:36:03 DEBUG 22:36:03 DEBUG 1686692163, 22:36:03 DEBUG 22:36:03 DEBUG "0" 22:36:03 DEBUG "pod": "ds-idrepo-0" 22:36:03 DEBUG ] 22:36:03 DEBUG }, 22:36:03 DEBUG } 22:36:03 DEBUG "value": [ 22:36:03 DEBUG ] 22:36:03 DEBUG 1686692163, 22:36:03 DEBUG } 22:36:03 DEBUG "605.1666666666666" 22:36:03 DEBUG } 22:36:03 DEBUG ] 22:36:03 DEBUG } 22:36:03 DEBUG ] 22:36:03 DEBUG } 22:36:03 DEBUG } 22:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692163" 22:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692163" 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG --- status code --- 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:04 DEBUG --- http response --- 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG { 22:36:04 DEBUG ----- output ----- 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG { 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG "data": { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "data": { 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG "result": [ 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:04 DEBUG { 22:36:04 DEBUG { 22:36:04 DEBUG { 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG "result": [ 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "metric": { 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG { 22:36:04 DEBUG { 22:36:04 DEBUG "data": { 22:36:04 DEBUG "data": { 22:36:04 DEBUG "pod": "ds-cts-0" 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG }, 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:04 DEBUG "pod": "ds-cts-0" 22:36:04 DEBUG "data": { 22:36:04 DEBUG "result": [ 22:36:04 DEBUG "result": [ 22:36:04 DEBUG "value": [ 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG }, 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG { 22:36:04 DEBUG { 22:36:04 DEBUG 22:36:04 DEBUG { 22:36:04 DEBUG "value": [ 22:36:04 DEBUG "result": [ 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG 22:36:04 DEBUG { 22:36:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:36:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:36:04 DEBUG 22:36:04 DEBUG "data": { 22:36:04 DEBUG 22:36:04 DEBUG "metric": { 22:36:04 DEBUG }, 22:36:04 DEBUG }, 22:36:04 DEBUG [...] 22:36:04 DEBUG "resultType": "vector", 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 DEBUG 22:36:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:36:04 DEBUG "value": [ 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG "result": [ 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG [...] 22:36:04 DEBUG }, 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG { 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "metric": { 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "pod": "ds-idrepo-0" 22:36:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG [...] 22:36:04 DEBUG }, 22:36:04 DEBUG }, 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:04 DEBUG "pod": "ds-idrepo-0" 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "value": [ 22:36:04 DEBUG "value": [ 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG }, 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 1686692163, 22:36:04 DEBUG 22:36:04 DEBUG { 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "605.1333333333333" 22:36:04 DEBUG 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG 1686692163, 22:36:04 DEBUG 22:36:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:36:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:36:04 DEBUG ] 22:36:04 DEBUG 22:36:04 DEBUG "data": { 22:36:04 DEBUG "0" 22:36:04 DEBUG 22:36:04 DEBUG }, 22:36:04 DEBUG }, 22:36:04 DEBUG } 22:36:04 DEBUG [...] 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG ] 22:36:04 DEBUG "pod": "idm-58dc667486-rgz9j" 22:36:04 DEBUG "value": [ 22:36:04 DEBUG "value": [ 22:36:04 DEBUG ] 22:36:04 DEBUG 22:36:04 DEBUG "result": [ 22:36:04 DEBUG } 22:36:04 DEBUG }, 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG } 22:36:04 DEBUG 22:36:04 DEBUG { 22:36:04 DEBUG ] 22:36:04 DEBUG "value": [ 22:36:04 DEBUG "0" 22:36:04 DEBUG "0" 22:36:04 DEBUG } 22:36:04 DEBUG 22:36:04 DEBUG "metric": { 22:36:04 DEBUG } 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG ] 22:36:04 DEBUG ] 22:36:04 DEBUG "pod": "idm-58dc667486-rgz9j" 22:36:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:36:04 DEBUG } 22:36:04 DEBUG "0" 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 DEBUG }, 22:36:04 DEBUG }, 22:36:04 DEBUG ] 22:36:04 DEBUG ] 22:36:04 DEBUG ] 22:36:04 DEBUG "value": [ 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG "value": [ 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 DEBUG "0" 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:36:04 DEBUG [...] 22:36:04 DEBUG ] 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG 22:36:04 DEBUG } 22:36:04 DEBUG { 22:36:04 DEBUG 22:36:04 DEBUG } 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG 22:36:04 DEBUG "data": { 22:36:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG }, 22:36:04 DEBUG "result": [ 22:36:04 DEBUG "value": [ 22:36:04 DEBUG { 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "0.2" 22:36:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:36:04 DEBUG ] 22:36:04 DEBUG }, 22:36:04 DEBUG } 22:36:04 DEBUG "value": [ 22:36:04 DEBUG ] 22:36:04 DEBUG 22:36:04 DEBUG } 22:36:04 DEBUG 22:36:04 DEBUG } 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "pod": "ds-idrepo-0" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "3489.7825624667726" 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692164" 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "data": { 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "result": [ 22:36:04 DEBUG { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "instance": "10.142.0.100:9100" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "instance": "10.142.0.99:9100" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "0" 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "data": { 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "result": [ 22:36:04 DEBUG { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "pod": "ds-cts-0" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "pod": "ds-cts-2" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "0" 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "data": { 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "result": [ 22:36:04 DEBUG { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "0" 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692164" 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "data": { 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "result": [ 22:36:04 DEBUG { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "13617078272" 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692164" 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "data": { 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "result": [ 22:36:04 DEBUG { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "instance": "10.142.0.100:9100" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "instance": "10.142.0.99:9100" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "0.00030000000000048505" 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "data": { 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "result": [ 22:36:04 DEBUG { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "pod": "ds-idrepo-2" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "pod": "ds-idrepo-0" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "605.1666666666666" 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692164" 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692164" 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG { 22:36:04 DEBUG { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "data": { 22:36:04 DEBUG "data": { 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "result": [ 22:36:04 DEBUG "result": [ 22:36:04 DEBUG { 22:36:04 DEBUG { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "instance": "10.142.0.100:9100" 22:36:04 DEBUG "instance": "10.142.0.100:9100" 22:36:04 DEBUG }, 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "instance": "10.142.0.99:9100" 22:36:04 DEBUG "instance": "10.142.0.99:9100" 22:36:04 DEBUG }, 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "19332053.8" 22:36:04 DEBUG "19328099.4" 22:36:04 DEBUG ] 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692164" 22:36:04 INFO [http_cmd]: http status code OK 22:36:04 DEBUG --- status code --- 22:36:04 DEBUG http status code is 200 (expected 200) 22:36:04 DEBUG --- http response --- 22:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:04 DEBUG ----- output ----- 22:36:04 DEBUG { 22:36:04 DEBUG "status": "success", 22:36:04 DEBUG "data": { 22:36:04 DEBUG "resultType": "vector", 22:36:04 DEBUG "result": [ 22:36:04 DEBUG { 22:36:04 DEBUG "metric": { 22:36:04 DEBUG "pod": "ds-idrepo-2" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG [...] 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG 22:36:04 DEBUG "pod": "ds-idrepo-0" 22:36:04 DEBUG }, 22:36:04 DEBUG "value": [ 22:36:04 DEBUG 1686692164, 22:36:04 DEBUG "605.1666666666666" 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG ] 22:36:04 DEBUG } 22:36:04 DEBUG } 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692165" 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected 200) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "pod": "ds-idrepo-0" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1686692165, 22:36:05 DEBUG "0.2531499202551834" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692165" 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected 200) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1686692165, 22:36:05 DEBUG "0.2316670361843429" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:21 INFO 22:36:21 INFO [loop_until]: kubectl --namespace=xlou top pods 22:36:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:36:21 INFO [loop_until]: OK (rc = 0) 22:36:21 DEBUG --- stdout --- 22:36:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 6m 5760Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 14m 425Mi ds-cts-2 9m 392Mi ds-idrepo-0 3130m 13803Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3000m 4946Mi idm-58dc667486-rgz9j 2420m 5097Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 211m 642Mi 22:36:21 DEBUG --- stderr --- 22:36:21 DEBUG 22:36:25 INFO 22:36:25 INFO [loop_until]: kubectl --namespace=xlou top node 22:36:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:36:25 INFO [loop_until]: OK (rc = 0) 22:36:25 DEBUG --- stdout --- 22:36:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3111m 19% 5888Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 71m 0% 6828Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2536m 15% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 450m 2% 3139Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1483Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 63m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 66m 0% 1204Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13724Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3406m 21% 14536Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 68m 0% 14603Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 271m 1% 1961Mi 3% 22:36:25 DEBUG --- stderr --- 22:36:25 DEBUG 22:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692223" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "ds-idrepo-2" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "ds-idrepo-0" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692223, 22:37:04 DEBUG "560.8" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0.2" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:37:04 DEBUG { 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG { 22:37:04 DEBUG "data": { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "result": [ 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG { 22:37:04 DEBUG "result": [ 22:37:04 DEBUG "metric": { 22:37:04 DEBUG { 22:37:04 DEBUG "pod": "ds-cts-0" 22:37:04 DEBUG "metric": { 22:37:04 DEBUG }, 22:37:04 DEBUG "pod": "ds-cts-2" 22:37:04 DEBUG "value": [ 22:37:04 DEBUG }, 22:37:04 DEBUG 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "ds-idrepo-0" 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 DEBUG 22:37:04 DEBUG }, 22:37:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:37:04 DEBUG "value": [ 22:37:04 DEBUG }, 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG "560.8" 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG ] 22:37:04 DEBUG "0.000029656013778584192" 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "ds-cts-0" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "ds-idrepo-0" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "ds-cts-2" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "ds-cts-2" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "instance": "10.142.0.100:9100" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "instance": "10.142.0.99:9100" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "ds-cts-0" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "ds-cts-2" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "13617586176" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "instance": "10.142.0.100:9100" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "instance": "10.142.0.99:9100" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "0.00026666666666604516" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "ds-idrepo-2" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "ds-idrepo-0" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "560.8333333333334" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692224" 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG { 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG "data": { 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "data": { 22:37:04 DEBUG "instance": "10.142.0.100:9100" 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG }, 22:37:04 DEBUG "result": [ 22:37:04 DEBUG "value": [ 22:37:04 DEBUG { 22:37:04 DEBUG 22:37:04 DEBUG "metric": { 22:37:04 DEBUG 22:37:04 DEBUG "instance": "10.142.0.100:9100" 22:37:04 DEBUG 22:37:04 DEBUG }, 22:37:04 DEBUG [...] 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "instance": "10.142.0.99:9100" 22:37:04 DEBUG [...] 22:37:04 DEBUG }, 22:37:04 DEBUG 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG 22:37:04 DEBUG "18933159.1" 22:37:04 DEBUG "instance": "10.142.0.99:9100" 22:37:04 DEBUG ] 22:37:04 DEBUG }, 22:37:04 DEBUG } 22:37:04 DEBUG "value": [ 22:37:04 DEBUG ] 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG } 22:37:04 DEBUG "18937052.766666666" 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692224" 22:37:04 INFO [http_cmd]: http status code OK 22:37:04 DEBUG --- status code --- 22:37:04 DEBUG http status code is 200 (expected 200) 22:37:04 DEBUG --- http response --- 22:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:04 DEBUG ----- output ----- 22:37:04 DEBUG { 22:37:04 DEBUG "status": "success", 22:37:04 DEBUG "data": { 22:37:04 DEBUG "resultType": "vector", 22:37:04 DEBUG "result": [ 22:37:04 DEBUG { 22:37:04 DEBUG "metric": { 22:37:04 DEBUG "pod": "ds-idrepo-2" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG [...] 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG 22:37:04 DEBUG "pod": "ds-idrepo-0" 22:37:04 DEBUG }, 22:37:04 DEBUG "value": [ 22:37:04 DEBUG 1686692224, 22:37:04 DEBUG "560.8666666666667" 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG ] 22:37:04 DEBUG } 22:37:04 DEBUG } 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692225" 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG http status code is 200 (expected 200) 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG { 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "data": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "pod": "ds-cts-2" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 1686692225, 22:37:05 DEBUG "0" 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692225" 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG http status code is 200 (expected 200) 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG { 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "data": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 1686692225, 22:37:05 DEBUG "0.23272943622234973" 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:22 INFO 22:37:22 INFO [loop_until]: kubectl --namespace=xlou top pods 22:37:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:37:22 INFO [loop_until]: OK (rc = 0) 22:37:22 DEBUG --- stdout --- 22:37:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 10m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 8m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3448m 13823Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 13m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2858m 4949Mi idm-58dc667486-rgz9j 2800m 5097Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 204m 643Mi 22:37:22 DEBUG --- stderr --- 22:37:22 DEBUG 22:37:25 INFO 22:37:25 INFO [loop_until]: kubectl --namespace=xlou top node 22:37:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:37:25 INFO [loop_until]: OK (rc = 0) 22:37:25 DEBUG --- stdout --- 22:37:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3053m 19% 5890Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 76m 0% 6829Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2916m 18% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 431m 2% 3138Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1205Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 74m 0% 13739Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3416m 21% 14531Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 66m 0% 1173Mi 2% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14603Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 273m 1% 1956Mi 3% 22:37:25 DEBUG --- stderr --- 22:37:25 DEBUG 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "ds-idrepo-2" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "ds-idrepo-0" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "581.7333333333333" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "0.2" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "0" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG --- http response --- 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG { 22:38:04 DEBUG { 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:04 DEBUG "data": { 22:38:04 DEBUG "data": { 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "result": [ 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG { 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "pod": "ds-cts-0" 22:38:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG }, 22:38:04 DEBUG }, 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "value": [ 22:38:04 DEBUG "value": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "ds-cts-0" 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:38:04 DEBUG }, 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG [...] 22:38:04 DEBUG [...] 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG "pod": "ds-idrepo-0" 22:38:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG }, 22:38:04 DEBUG }, 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "value": [ 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG 22:38:04 DEBUG "pod": "ds-idrepo-0" 22:38:04 DEBUG "0" 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG "0" 22:38:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:38:04 DEBUG }, 22:38:04 DEBUG ] 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG ] 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG } 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:04 DEBUG } 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG ] 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG ] 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "581.7333333333333" 22:38:04 DEBUG } 22:38:04 DEBUG { 22:38:04 DEBUG } 22:38:04 DEBUG "0" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG "data": { 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG "result": [ 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 DEBUG { 22:38:04 DEBUG } 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "0" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:38:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:38:04 DEBUG }, 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:38:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:38:04 DEBUG }, 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "4260.896702382191" 22:38:04 DEBUG "0.008196162973750122" 22:38:04 DEBUG ] 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "ds-cts-0" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "ds-cts-2" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "0" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "instance": "10.142.0.100:9100" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "instance": "10.142.0.99:9100" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "0.00033333333333333365" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "0" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "13619249152" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "instance": "10.142.0.100:9100" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "instance": "10.142.0.99:9100" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "0.0003333333333330302" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "ds-idrepo-2" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "ds-idrepo-0" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "581.7666666666667" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "instance": "10.142.0.100:9100" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "instance": "10.142.0.99:9100" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "17990066.933333334" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "instance": "10.142.0.100:9100" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "instance": "10.142.0.99:9100" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "17994235.933333334" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692284" 22:38:04 INFO [http_cmd]: http status code OK 22:38:04 DEBUG --- status code --- 22:38:04 DEBUG http status code is 200 (expected 200) 22:38:04 DEBUG --- http response --- 22:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:04 DEBUG ----- output ----- 22:38:04 DEBUG { 22:38:04 DEBUG "status": "success", 22:38:04 DEBUG "data": { 22:38:04 DEBUG "resultType": "vector", 22:38:04 DEBUG "result": [ 22:38:04 DEBUG { 22:38:04 DEBUG "metric": { 22:38:04 DEBUG "pod": "ds-idrepo-2" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG [...] 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG 22:38:04 DEBUG "pod": "ds-idrepo-0" 22:38:04 DEBUG }, 22:38:04 DEBUG "value": [ 22:38:04 DEBUG 1686692284, 22:38:04 DEBUG "581.7666666666667" 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG ] 22:38:04 DEBUG } 22:38:04 DEBUG } 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692285" 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692285" 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG http status code is 200 (expected 200) 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG { 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "data": { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "result": [ 22:38:05 DEBUG { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "pod": "ds-idrepo-2" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 1686692285, 22:38:05 DEBUG "0.6716736887097758" 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG http status code is 200 (expected 200) 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG { 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "data": { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "result": [ 22:38:05 DEBUG { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 1686692285, 22:38:05 DEBUG "0.224320731787457" 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:22 INFO 22:38:22 INFO [loop_until]: kubectl --namespace=xlou top pods 22:38:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:38:22 INFO [loop_until]: OK (rc = 0) 22:38:22 DEBUG --- stdout --- 22:38:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3244m 13802Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2885m 4950Mi idm-58dc667486-rgz9j 2943m 5097Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 206m 643Mi 22:38:22 DEBUG --- stderr --- 22:38:22 DEBUG 22:38:25 INFO 22:38:25 INFO [loop_until]: kubectl --namespace=xlou top node 22:38:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:38:25 INFO [loop_until]: OK (rc = 0) 22:38:25 DEBUG --- stdout --- 22:38:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2840m 17% 5892Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6863Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6825Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2949m 18% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 443m 2% 3132Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 78m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13722Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3419m 21% 14505Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14603Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 268m 1% 1959Mi 3% 22:38:25 DEBUG --- stderr --- 22:38:25 DEBUG 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:39:04 DEBUG "pod": "ds-idrepo-2" 22:39:04 DEBUG }, 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:39:04 DEBUG "pod": "ds-idrepo-0" 22:39:04 DEBUG }, 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "0.2" 22:39:04 DEBUG "631.6" 22:39:04 DEBUG ] 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG { 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG "data": { 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG "result": [ 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG { 22:39:04 DEBUG { 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:39:04 DEBUG "data": { 22:39:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:39:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:39:04 DEBUG }, 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG }, 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG "result": [ 22:39:04 DEBUG "value": [ 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG { 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "metric": { 22:39:04 DEBUG 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 DEBUG [...] 22:39:04 DEBUG }, 22:39:04 DEBUG [...] 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:39:04 DEBUG 22:39:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:39:04 DEBUG "pod": "ds-idrepo-0" 22:39:04 DEBUG }, 22:39:04 DEBUG [...] 22:39:04 DEBUG }, 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG "value": [ 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "0" 22:39:04 DEBUG 22:39:04 DEBUG "0" 22:39:04 DEBUG "2.0765329827547974" 22:39:04 DEBUG ] 22:39:04 DEBUG "pod": "ds-idrepo-0" 22:39:04 DEBUG ] 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG }, 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG "value": [ 22:39:04 DEBUG ] 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG "4027.126530464077" 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "pod": "ds-cts-0" 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:39:04 DEBUG }, 22:39:04 DEBUG "pod": "ds-cts-0" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "pod": "ds-idrepo-0" 22:39:04 DEBUG 22:39:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:39:04 DEBUG }, 22:39:04 DEBUG "pod": "ds-idrepo-0" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "631.6" 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "0" 22:39:04 DEBUG ] 22:39:04 DEBUG "0" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "pod": "ds-idrepo-0" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "778.9346808387061" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "instance": "10.142.0.100:9100" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "instance": "10.142.0.99:9100" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "0" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "pod": "ds-cts-0" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "pod": "ds-cts-2" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "0" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "0" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "13616021504" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "instance": "10.142.0.100:9100" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "instance": "10.142.0.99:9100" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "0.00033333333333492493" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "pod": "ds-idrepo-2" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "pod": "ds-idrepo-0" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "631.6" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "instance": "10.142.0.100:9100" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "instance": "10.142.0.99:9100" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "21481653.7" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "instance": "10.142.0.100:9100" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "instance": "10.142.0.99:9100" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "21485782.866666667" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692344" 22:39:04 INFO [http_cmd]: http status code OK 22:39:04 DEBUG --- status code --- 22:39:04 DEBUG http status code is 200 (expected 200) 22:39:04 DEBUG --- http response --- 22:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:04 DEBUG ----- output ----- 22:39:04 DEBUG { 22:39:04 DEBUG "status": "success", 22:39:04 DEBUG "data": { 22:39:04 DEBUG "resultType": "vector", 22:39:04 DEBUG "result": [ 22:39:04 DEBUG { 22:39:04 DEBUG "metric": { 22:39:04 DEBUG "pod": "ds-idrepo-2" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG [...] 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG 22:39:04 DEBUG "pod": "ds-idrepo-0" 22:39:04 DEBUG }, 22:39:04 DEBUG "value": [ 22:39:04 DEBUG 1686692344, 22:39:04 DEBUG "631.6333333333333" 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG ] 22:39:04 DEBUG } 22:39:04 DEBUG } 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692345" 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG http status code is 200 (expected 200) 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:39:05 DEBUG ----- output ----- 22:39:05 DEBUG { 22:39:05 DEBUG "status": "success", 22:39:05 DEBUG "data": { 22:39:05 DEBUG "resultType": "vector", 22:39:05 DEBUG "result": [ 22:39:05 DEBUG { 22:39:05 DEBUG "metric": { 22:39:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:39:05 DEBUG }, 22:39:05 DEBUG "value": [ 22:39:05 DEBUG 22:39:05 DEBUG 22:39:05 DEBUG 22:39:05 DEBUG [...] 22:39:05 DEBUG 22:39:05 DEBUG 22:39:05 DEBUG 22:39:05 DEBUG "pod": "ds-idrepo-0" 22:39:05 DEBUG }, 22:39:05 DEBUG "value": [ 22:39:05 DEBUG 1686692345, 22:39:05 DEBUG "0.2765420805018296" 22:39:05 DEBUG ] 22:39:05 DEBUG } 22:39:05 DEBUG ] 22:39:05 DEBUG } 22:39:05 DEBUG } 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692345" 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG http status code is 200 (expected 200) 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:05 DEBUG ----- output ----- 22:39:05 DEBUG { 22:39:05 DEBUG "status": "success", 22:39:05 DEBUG "data": { 22:39:05 DEBUG "resultType": "vector", 22:39:05 DEBUG "result": [ 22:39:05 DEBUG { 22:39:05 DEBUG "metric": { 22:39:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:39:05 DEBUG }, 22:39:05 DEBUG "value": [ 22:39:05 DEBUG 22:39:05 DEBUG 22:39:05 DEBUG 22:39:05 DEBUG [...] 22:39:05 DEBUG 22:39:05 DEBUG 22:39:05 DEBUG 22:39:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:39:05 DEBUG }, 22:39:05 DEBUG "value": [ 22:39:05 DEBUG 1686692345, 22:39:05 DEBUG "0.24132285924110075" 22:39:05 DEBUG ] 22:39:05 DEBUG } 22:39:05 DEBUG ] 22:39:05 DEBUG } 22:39:05 DEBUG } 22:39:22 INFO 22:39:22 INFO [loop_until]: kubectl --namespace=xlou top pods 22:39:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:39:22 INFO [loop_until]: OK (rc = 0) 22:39:22 DEBUG --- stdout --- 22:39:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 8m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3377m 13799Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3324m 4953Mi idm-58dc667486-rgz9j 2771m 5098Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 223m 643Mi 22:39:22 DEBUG --- stderr --- 22:39:22 DEBUG 22:39:25 INFO 22:39:25 INFO [loop_until]: kubectl --namespace=xlou top node 22:39:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:39:26 INFO [loop_until]: OK (rc = 0) 22:39:26 DEBUG --- stdout --- 22:39:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3419m 21% 5892Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6831Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2775m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 458m 2% 3132Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1201Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 67m 0% 13725Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3580m 22% 14504Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 77m 0% 14603Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 279m 1% 1961Mi 3% 22:39:26 DEBUG --- stderr --- 22:39:26 DEBUG 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "ds-idrepo-2" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "ds-idrepo-0" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "617.7333333333333" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "0.2" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG { 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG "data": { 22:40:04 DEBUG { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG "data": { 22:40:04 DEBUG { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "result": [ 22:40:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:40:04 DEBUG { 22:40:04 DEBUG }, 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "value": [ 22:40:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:40:04 DEBUG 22:40:04 DEBUG }, 22:40:04 DEBUG 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:40:04 DEBUG 22:40:04 DEBUG }, 22:40:04 DEBUG 22:40:04 DEBUG "value": [ 22:40:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG }, 22:40:04 DEBUG "0" 22:40:04 DEBUG "value": [ 22:40:04 DEBUG ] 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG } 22:40:04 DEBUG "0.008232834041983512" 22:40:04 DEBUG ] 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "0" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "0" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "0" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "ds-cts-0" 22:40:04 DEBUG "pod": "ds-cts-0" 22:40:04 DEBUG }, 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "ds-idrepo-0" 22:40:04 DEBUG "pod": "ds-idrepo-0" 22:40:04 DEBUG }, 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "617.7" 22:40:04 DEBUG "0" 22:40:04 DEBUG ] 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "4388.228598212194" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "instance": "10.142.0.100:9100" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "instance": "10.142.0.99:9100" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "0" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "ds-cts-0" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "ds-cts-2" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "0" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "0" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "13617848320" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "instance": "10.142.0.100:9100" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "instance": "10.142.0.99:9100" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "0.00026666666666604516" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "ds-idrepo-2" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "ds-idrepo-0" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "617.7" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "instance": "10.142.0.100:9100" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "instance": "10.142.0.99:9100" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "20099542.8" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "instance": "10.142.0.100:9100" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "instance": "10.142.0.99:9100" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "20104232.933333334" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692404" 22:40:04 INFO [http_cmd]: http status code OK 22:40:04 DEBUG --- status code --- 22:40:04 DEBUG http status code is 200 (expected 200) 22:40:04 DEBUG --- http response --- 22:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:04 DEBUG ----- output ----- 22:40:04 DEBUG { 22:40:04 DEBUG "status": "success", 22:40:04 DEBUG "data": { 22:40:04 DEBUG "resultType": "vector", 22:40:04 DEBUG "result": [ 22:40:04 DEBUG { 22:40:04 DEBUG "metric": { 22:40:04 DEBUG "pod": "ds-idrepo-2" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG [...] 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG 22:40:04 DEBUG "pod": "ds-idrepo-0" 22:40:04 DEBUG }, 22:40:04 DEBUG "value": [ 22:40:04 DEBUG 1686692404, 22:40:04 DEBUG "617.6666666666666" 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG ] 22:40:04 DEBUG } 22:40:04 DEBUG } 22:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692405" 22:40:05 INFO [http_cmd]: http status code OK 22:40:05 DEBUG --- status code --- 22:40:05 DEBUG http status code is 200 (expected 200) 22:40:05 DEBUG --- http response --- 22:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:40:05 DEBUG ----- output ----- 22:40:05 DEBUG { 22:40:05 DEBUG "status": "success", 22:40:05 DEBUG "data": { 22:40:05 DEBUG "resultType": "vector", 22:40:05 DEBUG "result": [ 22:40:05 DEBUG { 22:40:05 DEBUG "metric": { 22:40:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:40:05 DEBUG }, 22:40:05 DEBUG "value": [ 22:40:05 DEBUG 22:40:05 DEBUG 22:40:05 DEBUG 22:40:05 DEBUG [...] 22:40:05 DEBUG 22:40:05 DEBUG 22:40:05 DEBUG 22:40:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:40:05 DEBUG }, 22:40:05 DEBUG "value": [ 22:40:05 DEBUG 1686692405, 22:40:05 DEBUG "0" 22:40:05 DEBUG ] 22:40:05 DEBUG } 22:40:05 DEBUG ] 22:40:05 DEBUG } 22:40:05 DEBUG } 22:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692405" 22:40:05 INFO [http_cmd]: http status code OK 22:40:05 DEBUG --- status code --- 22:40:05 DEBUG http status code is 200 (expected 200) 22:40:05 DEBUG --- http response --- 22:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:05 DEBUG ----- output ----- 22:40:05 DEBUG { 22:40:05 DEBUG "status": "success", 22:40:05 DEBUG "data": { 22:40:05 DEBUG "resultType": "vector", 22:40:05 DEBUG "result": [ 22:40:05 DEBUG { 22:40:05 DEBUG "metric": { 22:40:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:40:05 DEBUG }, 22:40:05 DEBUG "value": [ 22:40:05 DEBUG 22:40:05 DEBUG 22:40:05 DEBUG 22:40:05 DEBUG [...] 22:40:05 DEBUG 22:40:05 DEBUG 22:40:05 DEBUG 22:40:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:40:05 DEBUG }, 22:40:05 DEBUG "value": [ 22:40:05 DEBUG 1686692405, 22:40:05 DEBUG "0.2329637955744699" 22:40:05 DEBUG ] 22:40:05 DEBUG } 22:40:05 DEBUG ] 22:40:05 DEBUG } 22:40:05 DEBUG } 22:40:22 INFO 22:40:22 INFO [loop_until]: kubectl --namespace=xlou top pods 22:40:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:40:22 INFO [loop_until]: OK (rc = 0) 22:40:22 DEBUG --- stdout --- 22:40:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3416m 13800Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2948m 4955Mi idm-58dc667486-rgz9j 2728m 5099Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 219m 642Mi 22:40:22 DEBUG --- stderr --- 22:40:22 DEBUG 22:40:26 INFO 22:40:26 INFO [loop_until]: kubectl --namespace=xlou top node 22:40:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:40:26 INFO [loop_until]: OK (rc = 0) 22:40:26 DEBUG --- stdout --- 22:40:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3224m 20% 5894Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6831Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2775m 17% 6075Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 437m 2% 3143Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13725Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3413m 21% 14509Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14604Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 289m 1% 1961Mi 3% 22:40:26 DEBUG --- stderr --- 22:40:26 DEBUG 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0.2" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "ds-idrepo-2" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "ds-idrepo-0" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "643.4666666666667" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:41:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:41:04 DEBUG }, 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:41:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:41:04 DEBUG }, 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0" 22:41:04 DEBUG "0.0031631963757587545" 22:41:04 DEBUG ] 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "ds-cts-0" 22:41:04 DEBUG "pod": "ds-cts-0" 22:41:04 DEBUG }, 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "ds-idrepo-0" 22:41:04 DEBUG "pod": "ds-idrepo-0" 22:41:04 DEBUG }, 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0" 22:41:04 DEBUG "643.4666666666667" 22:41:04 DEBUG ] 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "152540351.01618883" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "instance": "10.142.0.100:9100" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "instance": "10.142.0.99:9100" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "ds-cts-0" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "ds-cts-2" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "13616525312" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "instance": "10.142.0.100:9100" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "instance": "10.142.0.99:9100" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "0.00026666666666604516" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "pod": "ds-idrepo-2" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "pod": "ds-idrepo-0" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "643.4666666666667" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "instance": "10.142.0.100:9100" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "instance": "10.142.0.99:9100" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "20039179.366666667" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692464" 22:41:04 INFO [http_cmd]: http status code OK 22:41:04 DEBUG --- status code --- 22:41:04 DEBUG http status code is 200 (expected 200) 22:41:04 DEBUG --- http response --- 22:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:04 DEBUG ----- output ----- 22:41:04 DEBUG { 22:41:04 DEBUG "status": "success", 22:41:04 DEBUG "data": { 22:41:04 DEBUG "resultType": "vector", 22:41:04 DEBUG "result": [ 22:41:04 DEBUG { 22:41:04 DEBUG "metric": { 22:41:04 DEBUG "instance": "10.142.0.100:9100" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG [...] 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG 22:41:04 DEBUG "instance": "10.142.0.99:9100" 22:41:04 DEBUG }, 22:41:04 DEBUG "value": [ 22:41:04 DEBUG 1686692464, 22:41:04 DEBUG "20043032.766666666" 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG ] 22:41:04 DEBUG } 22:41:04 DEBUG } 22:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692465" 22:41:05 INFO [http_cmd]: http status code OK 22:41:05 DEBUG --- status code --- 22:41:05 DEBUG http status code is 200 (expected 200) 22:41:05 DEBUG --- http response --- 22:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:05 DEBUG ----- output ----- 22:41:05 DEBUG { 22:41:05 DEBUG "status": "success", 22:41:05 DEBUG "data": { 22:41:05 DEBUG "resultType": "vector", 22:41:05 DEBUG "result": [ 22:41:05 DEBUG { 22:41:05 DEBUG "metric": { 22:41:05 DEBUG "pod": "ds-idrepo-2" 22:41:05 DEBUG }, 22:41:05 DEBUG "value": [ 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG [...] 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG "pod": "ds-idrepo-0" 22:41:05 DEBUG }, 22:41:05 DEBUG "value": [ 22:41:05 DEBUG 1686692465, 22:41:05 DEBUG "643.4666666666667" 22:41:05 DEBUG ] 22:41:05 DEBUG } 22:41:05 DEBUG ] 22:41:05 DEBUG } 22:41:05 DEBUG } 22:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692465" 22:41:05 INFO [http_cmd]: http status code OK 22:41:05 DEBUG --- status code --- 22:41:05 DEBUG http status code is 200 (expected 200) 22:41:05 DEBUG --- http response --- 22:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:41:05 DEBUG ----- output ----- 22:41:05 DEBUG { 22:41:05 DEBUG "status": "success", 22:41:05 DEBUG "data": { 22:41:05 DEBUG "resultType": "vector", 22:41:05 DEBUG "result": [ 22:41:05 DEBUG { 22:41:05 DEBUG "metric": { 22:41:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:41:05 DEBUG }, 22:41:05 DEBUG "value": [ 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG [...] 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 22:41:05 DEBUG }, 22:41:05 DEBUG "value": [ 22:41:05 DEBUG 1686692465, 22:41:05 DEBUG "0" 22:41:05 DEBUG ] 22:41:05 DEBUG } 22:41:05 DEBUG ] 22:41:05 DEBUG } 22:41:05 DEBUG } 22:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692465" 22:41:05 INFO [http_cmd]: http status code OK 22:41:05 DEBUG --- status code --- 22:41:05 DEBUG http status code is 200 (expected 200) 22:41:05 DEBUG --- http response --- 22:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:05 DEBUG ----- output ----- 22:41:05 DEBUG { 22:41:05 DEBUG "status": "success", 22:41:05 DEBUG "data": { 22:41:05 DEBUG "resultType": "vector", 22:41:05 DEBUG "result": [ 22:41:05 DEBUG { 22:41:05 DEBUG "metric": { 22:41:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:41:05 DEBUG }, 22:41:05 DEBUG "value": [ 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG [...] 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG 22:41:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:41:05 DEBUG }, 22:41:05 DEBUG "value": [ 22:41:05 DEBUG 1686692465, 22:41:05 DEBUG "0.23548319185532302" 22:41:05 DEBUG ] 22:41:05 DEBUG } 22:41:05 DEBUG ] 22:41:05 DEBUG } 22:41:05 DEBUG } 22:41:22 INFO 22:41:22 INFO [loop_until]: kubectl --namespace=xlou top pods 22:41:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:41:22 INFO [loop_until]: OK (rc = 0) 22:41:22 DEBUG --- stdout --- 22:41:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 10m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3412m 13806Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3097m 4956Mi idm-58dc667486-rgz9j 3145m 5098Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 213m 643Mi 22:41:22 DEBUG --- stderr --- 22:41:22 DEBUG 22:41:26 INFO 22:41:26 INFO [loop_until]: kubectl --namespace=xlou top node 22:41:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:41:26 INFO [loop_until]: OK (rc = 0) 22:41:26 DEBUG --- stdout --- 22:41:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3229m 20% 5895Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 3079m 19% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 458m 2% 3131Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 65m 0% 1203Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13726Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3564m 22% 14515Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14599Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 283m 1% 1962Mi 3% 22:41:26 DEBUG --- stderr --- 22:41:26 DEBUG 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "ds-cts-2" 22:42:04 DEBUG "pod": "ds-idrepo-2" 22:42:04 DEBUG }, 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:42:04 DEBUG "pod": "ds-idrepo-0" 22:42:04 DEBUG }, 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG "623.0666666666666" 22:42:04 DEBUG ] 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0.2" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "ds-cts-2" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "ds-cts-2" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "ds-cts-0" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "ds-idrepo-0" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "623.0666666666666" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "ds-cts-0" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "ds-idrepo-0" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "instance": "10.142.0.100:9100" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "instance": "10.142.0.99:9100" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "ds-cts-0" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "ds-cts-2" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "13617778688" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "instance": "10.142.0.100:9100" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "instance": "10.142.0.99:9100" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "0.0002999999999985903" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "pod": "ds-idrepo-2" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "pod": "ds-idrepo-0" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "623.0666666666666" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692524" 22:42:04 INFO [http_cmd]: http status code OK 22:42:04 DEBUG --- status code --- 22:42:04 DEBUG http status code is 200 (expected 200) 22:42:04 DEBUG --- http response --- 22:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:04 DEBUG ----- output ----- 22:42:04 DEBUG { 22:42:04 DEBUG "status": "success", 22:42:04 DEBUG "data": { 22:42:04 DEBUG "resultType": "vector", 22:42:04 DEBUG "result": [ 22:42:04 DEBUG { 22:42:04 DEBUG "metric": { 22:42:04 DEBUG "instance": "10.142.0.100:9100" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG [...] 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG 22:42:04 DEBUG "instance": "10.142.0.99:9100" 22:42:04 DEBUG }, 22:42:04 DEBUG "value": [ 22:42:04 DEBUG 1686692524, 22:42:04 DEBUG "18912925.666666664" 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG ] 22:42:04 DEBUG } 22:42:04 DEBUG } 22:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692525" 22:42:05 INFO [http_cmd]: http status code OK 22:42:05 DEBUG --- status code --- 22:42:05 DEBUG http status code is 200 (expected 200) 22:42:05 DEBUG --- http response --- 22:42:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:05 DEBUG ----- output ----- 22:42:05 DEBUG { 22:42:05 DEBUG "status": "success", 22:42:05 DEBUG "data": { 22:42:05 DEBUG "resultType": "vector", 22:42:05 DEBUG "result": [ 22:42:05 DEBUG { 22:42:05 DEBUG "metric": { 22:42:05 DEBUG "instance": "10.142.0.100:9100" 22:42:05 DEBUG }, 22:42:05 DEBUG "value": [ 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG [...] 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG "instance": "10.142.0.99:9100" 22:42:05 DEBUG }, 22:42:05 DEBUG "value": [ 22:42:05 DEBUG 1686692525, 22:42:05 DEBUG "18917091" 22:42:05 DEBUG ] 22:42:05 DEBUG } 22:42:05 DEBUG ] 22:42:05 DEBUG } 22:42:05 DEBUG } 22:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692525" 22:42:05 INFO [http_cmd]: http status code OK 22:42:05 DEBUG --- status code --- 22:42:05 DEBUG http status code is 200 (expected 200) 22:42:05 DEBUG --- http response --- 22:42:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:05 DEBUG ----- output ----- 22:42:05 DEBUG { 22:42:05 DEBUG "status": "success", 22:42:05 DEBUG "data": { 22:42:05 DEBUG "resultType": "vector", 22:42:05 DEBUG "result": [ 22:42:05 DEBUG { 22:42:05 DEBUG "metric": { 22:42:05 DEBUG "pod": "ds-idrepo-2" 22:42:05 DEBUG }, 22:42:05 DEBUG "value": [ 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG [...] 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG "pod": "ds-idrepo-0" 22:42:05 DEBUG }, 22:42:05 DEBUG "value": [ 22:42:05 DEBUG 1686692525, 22:42:05 DEBUG "623.0333333333333" 22:42:05 DEBUG ] 22:42:05 DEBUG } 22:42:05 DEBUG ] 22:42:05 DEBUG } 22:42:05 DEBUG } 22:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692525" 22:42:05 INFO [http_cmd]: http status code OK 22:42:05 DEBUG --- status code --- 22:42:05 DEBUG http status code is 200 (expected 200) 22:42:05 DEBUG --- http response --- 22:42:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:42:05 DEBUG ----- output ----- 22:42:05 DEBUG { 22:42:05 DEBUG "status": "success", 22:42:05 DEBUG "data": { 22:42:05 DEBUG "resultType": "vector", 22:42:05 DEBUG "result": [ 22:42:05 DEBUG { 22:42:05 DEBUG "metric": { 22:42:05 DEBUG "pod": "ds-cts-2" 22:42:05 DEBUG }, 22:42:05 DEBUG "value": [ 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG [...] 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:42:05 DEBUG }, 22:42:05 DEBUG "value": [ 22:42:05 DEBUG 1686692525, 22:42:05 DEBUG "0" 22:42:05 DEBUG ] 22:42:05 DEBUG } 22:42:05 DEBUG ] 22:42:05 DEBUG } 22:42:05 DEBUG } 22:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692525" 22:42:05 INFO [http_cmd]: http status code OK 22:42:05 DEBUG --- status code --- 22:42:05 DEBUG http status code is 200 (expected 200) 22:42:05 DEBUG --- http response --- 22:42:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:05 DEBUG ----- output ----- 22:42:05 DEBUG { 22:42:05 DEBUG "status": "success", 22:42:05 DEBUG "data": { 22:42:05 DEBUG "resultType": "vector", 22:42:05 DEBUG "result": [ 22:42:05 DEBUG { 22:42:05 DEBUG "metric": { 22:42:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:42:05 DEBUG }, 22:42:05 DEBUG "value": [ 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG [...] 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG 22:42:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:42:05 DEBUG }, 22:42:05 DEBUG "value": [ 22:42:05 DEBUG 1686692525, 22:42:05 DEBUG "0.23608123758046307" 22:42:05 DEBUG ] 22:42:05 DEBUG } 22:42:05 DEBUG ] 22:42:05 DEBUG } 22:42:05 DEBUG } 22:42:22 INFO 22:42:22 INFO [loop_until]: kubectl --namespace=xlou top pods 22:42:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:42:22 INFO [loop_until]: OK (rc = 0) 22:42:22 DEBUG --- stdout --- 22:42:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 10m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3449m 13804Mi ds-idrepo-1 88m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3172m 4958Mi idm-58dc667486-rgz9j 2683m 5098Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 201m 643Mi 22:42:22 DEBUG --- stderr --- 22:42:22 DEBUG 22:42:26 INFO 22:42:26 INFO [loop_until]: kubectl --namespace=xlou top node 22:42:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:42:26 INFO [loop_until]: OK (rc = 0) 22:42:26 DEBUG --- stdout --- 22:42:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3043m 19% 5899Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2755m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 428m 2% 3120Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 65m 0% 1208Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 72m 0% 13724Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3316m 20% 14504Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 60m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 149m 0% 14605Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 270m 1% 1961Mi 3% 22:42:26 DEBUG --- stderr --- 22:42:26 DEBUG 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0.2" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "ds-cts-2" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0.000019862738869804062" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "ds-idrepo-2" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "ds-idrepo-0" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "649.9666666666667" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "ds-cts-2" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "ds-cts-2" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG { 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:04 DEBUG "data": { 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG { 22:43:04 DEBUG "result": [ 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG { 22:43:04 DEBUG "data": { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "pod": "ds-cts-0" 22:43:04 DEBUG "result": [ 22:43:04 DEBUG }, 22:43:04 DEBUG { 22:43:04 DEBUG "value": [ 22:43:04 DEBUG "metric": { 22:43:04 DEBUG 22:43:04 DEBUG "pod": "ds-cts-0" 22:43:04 DEBUG 22:43:04 DEBUG }, 22:43:04 DEBUG 22:43:04 DEBUG "value": [ 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG "pod": "ds-idrepo-0" 22:43:04 DEBUG 22:43:04 DEBUG }, 22:43:04 DEBUG 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "pod": "ds-idrepo-0" 22:43:04 DEBUG "0" 22:43:04 DEBUG }, 22:43:04 DEBUG ] 22:43:04 DEBUG "value": [ 22:43:04 DEBUG } 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG ] 22:43:04 DEBUG "649.9333333333333" 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "instance": "10.142.0.100:9100" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "instance": "10.142.0.99:9100" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "ds-cts-0" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "ds-cts-2" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "13614874624" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "instance": "10.142.0.100:9100" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "instance": "10.142.0.99:9100" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "0.00026666666666793997" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692584" 22:43:04 INFO [http_cmd]: http status code OK 22:43:04 DEBUG --- status code --- 22:43:04 DEBUG http status code is 200 (expected 200) 22:43:04 DEBUG --- http response --- 22:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:04 DEBUG ----- output ----- 22:43:04 DEBUG { 22:43:04 DEBUG "status": "success", 22:43:04 DEBUG "data": { 22:43:04 DEBUG "resultType": "vector", 22:43:04 DEBUG "result": [ 22:43:04 DEBUG { 22:43:04 DEBUG "metric": { 22:43:04 DEBUG "pod": "ds-idrepo-2" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG [...] 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG 22:43:04 DEBUG "pod": "ds-idrepo-0" 22:43:04 DEBUG }, 22:43:04 DEBUG "value": [ 22:43:04 DEBUG 1686692584, 22:43:04 DEBUG "649.9333333333333" 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG ] 22:43:04 DEBUG } 22:43:04 DEBUG } 22:43:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692585" 22:43:05 INFO [http_cmd]: http status code OK 22:43:05 DEBUG --- status code --- 22:43:05 DEBUG http status code is 200 (expected 200) 22:43:05 DEBUG --- http response --- 22:43:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:05 DEBUG ----- output ----- 22:43:05 DEBUG { 22:43:05 DEBUG "status": "success", 22:43:05 DEBUG "data": { 22:43:05 DEBUG "resultType": "vector", 22:43:05 DEBUG "result": [ 22:43:05 DEBUG { 22:43:05 DEBUG "metric": { 22:43:05 DEBUG "instance": "10.142.0.100:9100" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG [...] 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG "instance": "10.142.0.99:9100" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 1686692585, 22:43:05 DEBUG "20378543.433333334" 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG } 22:43:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692585" 22:43:05 INFO [http_cmd]: http status code OK 22:43:05 DEBUG --- status code --- 22:43:05 DEBUG http status code is 200 (expected 200) 22:43:05 DEBUG --- http response --- 22:43:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:05 DEBUG ----- output ----- 22:43:05 DEBUG { 22:43:05 DEBUG "status": "success", 22:43:05 DEBUG "data": { 22:43:05 DEBUG "resultType": "vector", 22:43:05 DEBUG "result": [ 22:43:05 DEBUG { 22:43:05 DEBUG "metric": { 22:43:05 DEBUG "instance": "10.142.0.100:9100" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG [...] 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG "instance": "10.142.0.99:9100" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 1686692585, 22:43:05 DEBUG "20382424.233333334" 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG } 22:43:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692585" 22:43:05 INFO [http_cmd]: http status code OK 22:43:05 DEBUG --- status code --- 22:43:05 DEBUG http status code is 200 (expected 200) 22:43:05 DEBUG --- http response --- 22:43:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:05 DEBUG ----- output ----- 22:43:05 DEBUG { 22:43:05 DEBUG "status": "success", 22:43:05 DEBUG "data": { 22:43:05 DEBUG "resultType": "vector", 22:43:05 DEBUG "result": [ 22:43:05 DEBUG { 22:43:05 DEBUG "metric": { 22:43:05 DEBUG "pod": "ds-idrepo-2" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG [...] 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG "pod": "ds-idrepo-0" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 1686692585, 22:43:05 DEBUG "649.9" 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG } 22:43:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692585" 22:43:05 INFO [http_cmd]: http status code OK 22:43:05 DEBUG --- status code --- 22:43:05 DEBUG http status code is 200 (expected 200) 22:43:05 DEBUG --- http response --- 22:43:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:43:05 DEBUG ----- output ----- 22:43:05 DEBUG { 22:43:05 DEBUG "status": "success", 22:43:05 DEBUG "data": { 22:43:05 DEBUG "resultType": "vector", 22:43:05 DEBUG "result": [ 22:43:05 DEBUG { 22:43:05 DEBUG "metric": { 22:43:05 DEBUG "pod": "ds-cts-2" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG [...] 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 1686692585, 22:43:05 DEBUG "0" 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG } 22:43:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692585" 22:43:05 INFO [http_cmd]: http status code OK 22:43:05 DEBUG --- status code --- 22:43:05 DEBUG http status code is 200 (expected 200) 22:43:05 DEBUG --- http response --- 22:43:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:05 DEBUG ----- output ----- 22:43:05 DEBUG { 22:43:05 DEBUG "status": "success", 22:43:05 DEBUG "data": { 22:43:05 DEBUG "resultType": "vector", 22:43:05 DEBUG "result": [ 22:43:05 DEBUG { 22:43:05 DEBUG "metric": { 22:43:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG [...] 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG 22:43:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:43:05 DEBUG }, 22:43:05 DEBUG "value": [ 22:43:05 DEBUG 1686692585, 22:43:05 DEBUG "0.23968149916441525" 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG ] 22:43:05 DEBUG } 22:43:05 DEBUG } 22:43:22 INFO 22:43:22 INFO [loop_until]: kubectl --namespace=xlou top pods 22:43:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:22 INFO [loop_until]: OK (rc = 0) 22:43:22 DEBUG --- stdout --- 22:43:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 7m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3702m 13801Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3296m 4961Mi idm-58dc667486-rgz9j 2862m 5099Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 213m 643Mi 22:43:22 DEBUG --- stderr --- 22:43:22 DEBUG 22:43:26 INFO 22:43:26 INFO [loop_until]: kubectl --namespace=xlou top node 22:43:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:26 INFO [loop_until]: OK (rc = 0) 22:43:26 DEBUG --- stdout --- 22:43:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3382m 21% 5896Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6865Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6828Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2958m 18% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 457m 2% 3130Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 73m 0% 13727Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3772m 23% 14507Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14606Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 285m 1% 1964Mi 3% 22:43:26 DEBUG --- stderr --- 22:43:26 DEBUG 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "ds-cts-2" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:44:04 DEBUG "pod": "ds-idrepo-2" 22:44:04 DEBUG }, 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:44:04 DEBUG "pod": "ds-idrepo-0" 22:44:04 DEBUG }, 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0.2" 22:44:04 DEBUG "609.1" 22:44:04 DEBUG ] 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "ds-cts-2" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "ds-cts-0" 22:44:04 DEBUG "pod": "ds-cts-2" 22:44:04 DEBUG }, 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "ds-idrepo-0" 22:44:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:44:04 DEBUG }, 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "609.1333333333333" 22:44:04 DEBUG "0" 22:44:04 DEBUG ] 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "ds-cts-0" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "ds-idrepo-0" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692644" 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "instance": "10.142.0.100:9100" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "instance": "10.142.0.99:9100" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0.00033333333333333365" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "ds-cts-0" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "ds-cts-2" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692644" 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692644" 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692644" 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "13616132096" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:04 INFO [http_cmd]: http status code OK 22:44:04 DEBUG --- status code --- 22:44:04 DEBUG http status code is 200 (expected 200) 22:44:04 DEBUG --- http response --- 22:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:04 DEBUG ----- output ----- 22:44:04 DEBUG { 22:44:04 DEBUG "status": "success", 22:44:04 DEBUG "data": { 22:44:04 DEBUG "resultType": "vector", 22:44:04 DEBUG "result": [ 22:44:04 DEBUG { 22:44:04 DEBUG "metric": { 22:44:04 DEBUG "instance": "10.142.0.100:9100" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG [...] 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG 22:44:04 DEBUG "instance": "10.142.0.99:9100" 22:44:04 DEBUG }, 22:44:04 DEBUG "value": [ 22:44:04 DEBUG 1686692644, 22:44:04 DEBUG "0.00040000000000001514" 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG ] 22:44:04 DEBUG } 22:44:04 DEBUG } 22:44:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692645" 22:44:05 INFO [http_cmd]: http status code OK 22:44:05 DEBUG --- status code --- 22:44:05 DEBUG http status code is 200 (expected 200) 22:44:05 DEBUG --- http response --- 22:44:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:05 DEBUG ----- output ----- 22:44:05 DEBUG { 22:44:05 DEBUG "status": "success", 22:44:05 DEBUG "data": { 22:44:05 DEBUG "resultType": "vector", 22:44:05 DEBUG "result": [ 22:44:05 DEBUG { 22:44:05 DEBUG "metric": { 22:44:05 DEBUG "pod": "ds-idrepo-2" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG [...] 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG "pod": "ds-idrepo-0" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 1686692645, 22:44:05 DEBUG "609.1333333333333" 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG } 22:44:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692645" 22:44:05 INFO [http_cmd]: http status code OK 22:44:05 DEBUG --- status code --- 22:44:05 DEBUG http status code is 200 (expected 200) 22:44:05 DEBUG --- http response --- 22:44:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:05 DEBUG ----- output ----- 22:44:05 DEBUG { 22:44:05 DEBUG "status": "success", 22:44:05 DEBUG "data": { 22:44:05 DEBUG "resultType": "vector", 22:44:05 DEBUG "result": [ 22:44:05 DEBUG { 22:44:05 DEBUG "metric": { 22:44:05 DEBUG "instance": "10.142.0.100:9100" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG [...] 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG "instance": "10.142.0.99:9100" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 1686692645, 22:44:05 DEBUG "18700500.833333336" 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG } 22:44:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692645" 22:44:05 INFO [http_cmd]: http status code OK 22:44:05 DEBUG --- status code --- 22:44:05 DEBUG http status code is 200 (expected 200) 22:44:05 DEBUG --- http response --- 22:44:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:05 DEBUG ----- output ----- 22:44:05 DEBUG { 22:44:05 DEBUG "status": "success", 22:44:05 DEBUG "data": { 22:44:05 DEBUG "resultType": "vector", 22:44:05 DEBUG "result": [ 22:44:05 DEBUG { 22:44:05 DEBUG "metric": { 22:44:05 DEBUG "instance": "10.142.0.100:9100" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG [...] 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG "instance": "10.142.0.99:9100" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 1686692645, 22:44:05 DEBUG "18704648.966666665" 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG } 22:44:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692645" 22:44:05 INFO [http_cmd]: http status code OK 22:44:05 DEBUG --- status code --- 22:44:05 DEBUG http status code is 200 (expected 200) 22:44:05 DEBUG --- http response --- 22:44:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:05 DEBUG ----- output ----- 22:44:05 DEBUG { 22:44:05 DEBUG "status": "success", 22:44:05 DEBUG "data": { 22:44:05 DEBUG "resultType": "vector", 22:44:05 DEBUG "result": [ 22:44:05 DEBUG { 22:44:05 DEBUG "metric": { 22:44:05 DEBUG "pod": "ds-idrepo-2" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG [...] 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG "pod": "ds-idrepo-0" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 1686692645, 22:44:05 DEBUG "609.1333333333333" 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG } 22:44:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692645" 22:44:05 INFO [http_cmd]: http status code OK 22:44:05 DEBUG --- status code --- 22:44:05 DEBUG http status code is 200 (expected 200) 22:44:05 DEBUG --- http response --- 22:44:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:44:05 DEBUG ----- output ----- 22:44:05 DEBUG { 22:44:05 DEBUG "status": "success", 22:44:05 DEBUG "data": { 22:44:05 DEBUG "resultType": "vector", 22:44:05 DEBUG "result": [ 22:44:05 DEBUG { 22:44:05 DEBUG "metric": { 22:44:05 DEBUG "pod": "ds-cts-2" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG [...] 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG "pod": "overseer-0-7bb59f88f9-gcrrq" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 1686692645, 22:44:05 DEBUG "0" 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG } 22:44:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692645" 22:44:05 INFO [http_cmd]: http status code OK 22:44:05 DEBUG --- status code --- 22:44:05 DEBUG http status code is 200 (expected 200) 22:44:05 DEBUG --- http response --- 22:44:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:05 DEBUG ----- output ----- 22:44:05 DEBUG { 22:44:05 DEBUG "status": "success", 22:44:05 DEBUG "data": { 22:44:05 DEBUG "resultType": "vector", 22:44:05 DEBUG "result": [ 22:44:05 DEBUG { 22:44:05 DEBUG "metric": { 22:44:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG [...] 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG 22:44:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:44:05 DEBUG }, 22:44:05 DEBUG "value": [ 22:44:05 DEBUG 1686692645, 22:44:05 DEBUG "0.24039599244648543" 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG ] 22:44:05 DEBUG } 22:44:05 DEBUG } 22:44:22 INFO 22:44:22 INFO [loop_until]: kubectl --namespace=xlou top pods 22:44:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:44:22 INFO [loop_until]: OK (rc = 0) 22:44:22 DEBUG --- stdout --- 22:44:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3133m 13821Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2852m 4963Mi idm-58dc667486-rgz9j 2522m 5098Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 193m 643Mi 22:44:22 DEBUG --- stderr --- 22:44:22 DEBUG 22:44:26 INFO 22:44:26 INFO [loop_until]: kubectl --namespace=xlou top node 22:44:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:44:26 INFO [loop_until]: OK (rc = 0) 22:44:26 DEBUG --- stdout --- 22:44:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2960m 18% 5900Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6863Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6829Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2590m 16% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 426m 2% 3126Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 85m 0% 1492Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1173Mi 2% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13725Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3192m 20% 14512Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14610Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 262m 1% 1959Mi 3% 22:44:26 DEBUG --- stderr --- 22:44:26 DEBUG 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG { 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "data": { 22:45:04 DEBUG "data": { 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "result": [ 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "pod": "ds-idrepo-2" 22:45:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:45:04 DEBUG }, 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG "value": [ 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [...] 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG { 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG 22:45:04 DEBUG "pod": "ds-idrepo-0" 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:45:04 DEBUG }, 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:04 DEBUG "data": { 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG { 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "582.4666666666667" 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "0" 22:45:04 DEBUG ] 22:45:04 DEBUG "data": { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "data": { 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG "result": [ 22:45:04 DEBUG }, 22:45:04 DEBUG "result": [ 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG { 22:45:04 DEBUG "value": [ 22:45:04 DEBUG { 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 DEBUG "metric": { 22:45:04 DEBUG 22:45:04 DEBUG "metric": { 22:45:04 DEBUG } 22:45:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:45:04 DEBUG 22:45:04 DEBUG "pod": "ds-cts-2" 22:45:04 DEBUG }, 22:45:04 DEBUG 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG [...] 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG [...] 22:45:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 DEBUG }, 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG 22:45:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:45:04 DEBUG "0" 22:45:04 DEBUG "pod": "ds-idrepo-0" 22:45:04 DEBUG }, 22:45:04 DEBUG ] 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG } 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG ] 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "0.2" 22:45:04 DEBUG } 22:45:04 DEBUG "0" 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "data": { 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "0" 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "data": { 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "pod": "ds-cts-2" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "0" 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "data": { 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "pod": "ds-cts-2" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG "pod": "ds-idrepo-0" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "0" 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "data": { 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "pod": "ds-cts-0" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG "pod": "ds-idrepo-0" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "582.4666666666667" 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "data": { 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "pod": "ds-cts-0" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG "pod": "ds-idrepo-0" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "0" 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692704" 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "data": { 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "instance": "10.142.0.100:9100" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG "instance": "10.142.0.99:9100" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "0" 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "data": { 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "pod": "ds-cts-0" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG "pod": "ds-cts-2" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "0" 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692704" 22:45:04 INFO [http_cmd]: http status code OK 22:45:04 DEBUG --- status code --- 22:45:04 DEBUG http status code is 200 (expected 200) 22:45:04 DEBUG --- http response --- 22:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:04 DEBUG ----- output ----- 22:45:04 DEBUG { 22:45:04 DEBUG "status": "success", 22:45:04 DEBUG "data": { 22:45:04 DEBUG "resultType": "vector", 22:45:04 DEBUG "result": [ 22:45:04 DEBUG { 22:45:04 DEBUG "metric": { 22:45:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG [...] 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG 22:45:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:45:04 DEBUG }, 22:45:04 DEBUG "value": [ 22:45:04 DEBUG 1686692704, 22:45:04 DEBUG "0" 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG ] 22:45:04 DEBUG } 22:45:04 DEBUG } 22:45:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692705" 22:45:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692705" 22:45:05 INFO [http_cmd]: http status code OK 22:45:05 DEBUG --- status code --- 22:45:05 DEBUG http status code is 200 (expected 200) 22:45:05 DEBUG --- http response --- 22:45:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:05 DEBUG ----- output ----- 22:45:05 DEBUG { 22:45:05 DEBUG "status": "success", 22:45:05 DEBUG "data": { 22:45:05 DEBUG "resultType": "vector", 22:45:05 DEBUG "result": [ 22:45:05 DEBUG { 22:45:05 DEBUG "metric": { 22:45:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG [...] 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 1686692705, 22:45:05 DEBUG "13616750592" 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG } 22:45:05 INFO [http_cmd]: http status code OK 22:45:05 DEBUG --- status code --- 22:45:05 DEBUG http status code is 200 (expected 200) 22:45:05 DEBUG --- http response --- 22:45:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:05 DEBUG ----- output ----- 22:45:05 DEBUG { 22:45:05 DEBUG "status": "success", 22:45:05 DEBUG "data": { 22:45:05 DEBUG "resultType": "vector", 22:45:05 DEBUG "result": [ 22:45:05 DEBUG { 22:45:05 DEBUG "metric": { 22:45:05 DEBUG "instance": "10.142.0.100:9100" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG [...] 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG "instance": "10.142.0.99:9100" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 1686692705, 22:45:05 DEBUG "0.00023333333333350008" 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG } 22:45:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692705" 22:45:05 INFO [http_cmd]: http status code OK 22:45:05 DEBUG --- status code --- 22:45:05 DEBUG http status code is 200 (expected 200) 22:45:05 DEBUG --- http response --- 22:45:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:05 DEBUG ----- output ----- 22:45:05 DEBUG { 22:45:05 DEBUG "status": "success", 22:45:05 DEBUG "data": { 22:45:05 DEBUG "resultType": "vector", 22:45:05 DEBUG "result": [ 22:45:05 DEBUG { 22:45:05 DEBUG "metric": { 22:45:05 DEBUG "pod": "ds-idrepo-2" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG [...] 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG "pod": "ds-idrepo-0" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 1686692705, 22:45:05 DEBUG "582.4666666666667" 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG } 22:45:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692705" 22:45:05 INFO [http_cmd]: http status code OK 22:45:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692705" 22:45:05 DEBUG --- status code --- 22:45:05 DEBUG http status code is 200 (expected 200) 22:45:05 DEBUG --- http response --- 22:45:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:05 DEBUG ----- output ----- 22:45:05 DEBUG { 22:45:05 DEBUG "status": "success", 22:45:05 DEBUG "data": { 22:45:05 DEBUG "resultType": "vector", 22:45:05 DEBUG "result": [ 22:45:05 DEBUG { 22:45:05 DEBUG "metric": { 22:45:05 DEBUG "instance": "10.142.0.100:9100" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG [...] 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG "instance": "10.142.0.99:9100" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 1686692705, 22:45:05 DEBUG "20022341.066666666" 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG } 22:45:05 INFO [http_cmd]: http status code OK 22:45:05 DEBUG --- status code --- 22:45:05 DEBUG http status code is 200 (expected 200) 22:45:05 DEBUG --- http response --- 22:45:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:05 DEBUG ----- output ----- 22:45:05 DEBUG { 22:45:05 DEBUG "status": "success", 22:45:05 DEBUG "data": { 22:45:05 DEBUG "resultType": "vector", 22:45:05 DEBUG "result": [ 22:45:05 DEBUG { 22:45:05 DEBUG "metric": { 22:45:05 DEBUG "instance": "10.142.0.100:9100" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG [...] 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG "instance": "10.142.0.99:9100" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 1686692705, 22:45:05 DEBUG "20026453.9" 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG } 22:45:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692705" 22:45:05 INFO [http_cmd]: http status code OK 22:45:05 DEBUG --- status code --- 22:45:05 DEBUG http status code is 200 (expected 200) 22:45:05 DEBUG --- http response --- 22:45:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:05 DEBUG ----- output ----- 22:45:05 DEBUG { 22:45:05 DEBUG "status": "success", 22:45:05 DEBUG "data": { 22:45:05 DEBUG "resultType": "vector", 22:45:05 DEBUG "result": [ 22:45:05 DEBUG { 22:45:05 DEBUG "metric": { 22:45:05 DEBUG "pod": "ds-idrepo-2" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG [...] 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG "pod": "ds-idrepo-0" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 1686692705, 22:45:05 DEBUG "582.4666666666667" 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG } 22:45:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692705" 22:45:05 INFO [http_cmd]: http status code OK 22:45:05 DEBUG --- status code --- 22:45:05 DEBUG http status code is 200 (expected 200) 22:45:05 DEBUG --- http response --- 22:45:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:45:05 DEBUG ----- output ----- 22:45:05 DEBUG { 22:45:05 DEBUG "status": "success", 22:45:05 DEBUG "data": { 22:45:05 DEBUG "resultType": "vector", 22:45:05 DEBUG "result": [ 22:45:05 DEBUG { 22:45:05 DEBUG "metric": { 22:45:05 DEBUG "pod": "ds-cts-2" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG [...] 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 1686692705, 22:45:05 DEBUG "0" 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG } 22:45:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692705" 22:45:05 INFO [http_cmd]: http status code OK 22:45:05 DEBUG --- status code --- 22:45:05 DEBUG http status code is 200 (expected 200) 22:45:05 DEBUG --- http response --- 22:45:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:05 DEBUG ----- output ----- 22:45:05 DEBUG { 22:45:05 DEBUG "status": "success", 22:45:05 DEBUG "data": { 22:45:05 DEBUG "resultType": "vector", 22:45:05 DEBUG "result": [ 22:45:05 DEBUG { 22:45:05 DEBUG "metric": { 22:45:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG [...] 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG 22:45:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:45:05 DEBUG }, 22:45:05 DEBUG "value": [ 22:45:05 DEBUG 1686692705, 22:45:05 DEBUG "0.225732194958171" 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG ] 22:45:05 DEBUG } 22:45:05 DEBUG } 22:45:22 INFO 22:45:22 INFO [loop_until]: kubectl --namespace=xlou top pods 22:45:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:45:22 INFO [loop_until]: OK (rc = 0) 22:45:22 DEBUG --- stdout --- 22:45:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 8m 417Mi ds-cts-1 8m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3220m 13810Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3398m 4964Mi idm-58dc667486-rgz9j 2495m 5099Mi lodemon-6455d5ff7c-zzndq 3m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 221m 644Mi 22:45:22 DEBUG --- stderr --- 22:45:22 DEBUG 22:45:26 INFO 22:45:26 INFO [loop_until]: kubectl --namespace=xlou top node 22:45:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:45:26 INFO [loop_until]: OK (rc = 0) 22:45:26 DEBUG --- stdout --- 22:45:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3304m 20% 5900Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6829Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2514m 15% 6081Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6952Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 448m 2% 3122Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 63m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 65m 0% 1201Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13726Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3437m 21% 14531Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14609Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 278m 1% 1964Mi 3% 22:45:26 DEBUG --- stderr --- 22:45:26 DEBUG 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG --- status code --- 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:46:04 DEBUG { 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG { 22:46:04 DEBUG "data": { 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "data": { 22:46:04 DEBUG "result": [ 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG { 22:46:04 DEBUG "result": [ 22:46:04 DEBUG "metric": { 22:46:04 DEBUG { 22:46:04 DEBUG "pod": "ds-idrepo-2" 22:46:04 DEBUG "metric": { 22:46:04 DEBUG }, 22:46:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:46:04 DEBUG "value": [ 22:46:04 DEBUG }, 22:46:04 DEBUG 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG "pod": "ds-idrepo-0" 22:46:04 DEBUG 22:46:04 DEBUG }, 22:46:04 DEBUG "pod": "ds-idrepo-0" 22:46:04 DEBUG "value": [ 22:46:04 DEBUG }, 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG "583.3333333333334" 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG ] 22:46:04 DEBUG "1.8649707420277852" 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG --- status code --- 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:04 DEBUG { 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG { 22:46:04 DEBUG { 22:46:04 DEBUG "data": { 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "data": { 22:46:04 DEBUG "data": { 22:46:04 DEBUG "result": [ 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG { 22:46:04 DEBUG "result": [ 22:46:04 DEBUG "result": [ 22:46:04 DEBUG "metric": { 22:46:04 DEBUG { 22:46:04 DEBUG { 22:46:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:46:04 DEBUG "metric": { 22:46:04 DEBUG "metric": { 22:46:04 DEBUG }, 22:46:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:46:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:46:04 DEBUG "value": [ 22:46:04 DEBUG }, 22:46:04 DEBUG }, 22:46:04 DEBUG 22:46:04 DEBUG "value": [ 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG }, 22:46:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:46:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:46:04 DEBUG "value": [ 22:46:04 DEBUG }, 22:46:04 DEBUG }, 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG "value": [ 22:46:04 DEBUG "0.2" 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG ] 22:46:04 DEBUG "0" 22:46:04 DEBUG "0" 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG ] 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG { 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "data": { 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "result": [ 22:46:04 DEBUG { 22:46:04 DEBUG "metric": { 22:46:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG "0" 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG --- status code --- 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:46:04 DEBUG { 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG { 22:46:04 DEBUG "data": { 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "data": { 22:46:04 DEBUG "result": [ 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG { 22:46:04 DEBUG "result": [ 22:46:04 DEBUG "metric": { 22:46:04 DEBUG { 22:46:04 DEBUG "pod": "ds-cts-0" 22:46:04 DEBUG "metric": { 22:46:04 DEBUG }, 22:46:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:46:04 DEBUG "value": [ 22:46:04 DEBUG }, 22:46:04 DEBUG 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG "pod": "ds-idrepo-0" 22:46:04 DEBUG 22:46:04 DEBUG }, 22:46:04 DEBUG "pod": "ds-idrepo-0" 22:46:04 DEBUG "value": [ 22:46:04 DEBUG }, 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG "583.3333333333334" 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG ] 22:46:04 DEBUG "3476.2308954537616" 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG { 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "data": { 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "result": [ 22:46:04 DEBUG { 22:46:04 DEBUG "metric": { 22:46:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG "pod": "ds-idrepo-0" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG "433434142.3215587" 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG { 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "data": { 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "result": [ 22:46:04 DEBUG { 22:46:04 DEBUG "metric": { 22:46:04 DEBUG "pod": "ds-cts-0" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG "pod": "ds-idrepo-0" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG "0" 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692764" 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG { 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "data": { 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "result": [ 22:46:04 DEBUG { 22:46:04 DEBUG "metric": { 22:46:04 DEBUG "instance": "10.142.0.100:9100" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG "instance": "10.142.0.99:9100" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG "0" 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG { 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "data": { 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "result": [ 22:46:04 DEBUG { 22:46:04 DEBUG "metric": { 22:46:04 DEBUG "pod": "ds-cts-0" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG "pod": "ds-cts-2" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG "0" 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692764" 22:46:04 INFO [http_cmd]: http status code OK 22:46:04 DEBUG --- status code --- 22:46:04 DEBUG http status code is 200 (expected 200) 22:46:04 DEBUG --- http response --- 22:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:04 DEBUG ----- output ----- 22:46:04 DEBUG { 22:46:04 DEBUG "status": "success", 22:46:04 DEBUG "data": { 22:46:04 DEBUG "resultType": "vector", 22:46:04 DEBUG "result": [ 22:46:04 DEBUG { 22:46:04 DEBUG "metric": { 22:46:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG [...] 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG 22:46:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:46:04 DEBUG }, 22:46:04 DEBUG "value": [ 22:46:04 DEBUG 1686692764, 22:46:04 DEBUG "0" 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG ] 22:46:04 DEBUG } 22:46:04 DEBUG } 22:46:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692765" 22:46:05 INFO [http_cmd]: http status code OK 22:46:05 DEBUG --- status code --- 22:46:05 DEBUG http status code is 200 (expected 200) 22:46:05 DEBUG --- http response --- 22:46:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:05 DEBUG ----- output ----- 22:46:05 DEBUG { 22:46:05 DEBUG "status": "success", 22:46:05 DEBUG "data": { 22:46:05 DEBUG "resultType": "vector", 22:46:05 DEBUG "result": [ 22:46:05 DEBUG { 22:46:05 DEBUG "metric": { 22:46:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG [...] 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 1686692765, 22:46:05 DEBUG "13618081792" 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG } 22:46:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692765" 22:46:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692765" 22:46:05 INFO [http_cmd]: http status code OK 22:46:05 DEBUG --- status code --- 22:46:05 DEBUG http status code is 200 (expected 200) 22:46:05 DEBUG --- http response --- 22:46:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:05 DEBUG ----- output ----- 22:46:05 DEBUG { 22:46:05 DEBUG "status": "success", 22:46:05 DEBUG "data": { 22:46:05 DEBUG "resultType": "vector", 22:46:05 DEBUG "result": [ 22:46:05 DEBUG { 22:46:05 DEBUG "metric": { 22:46:05 DEBUG "instance": "10.142.0.100:9100" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG [...] 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG "instance": "10.142.0.99:9100" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 1686692765, 22:46:05 DEBUG "0.00036666666666747006" 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG } 22:46:05 INFO [http_cmd]: http status code OK 22:46:05 DEBUG --- status code --- 22:46:05 DEBUG http status code is 200 (expected 200) 22:46:05 DEBUG --- http response --- 22:46:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:05 DEBUG ----- output ----- 22:46:05 DEBUG { 22:46:05 DEBUG "status": "success", 22:46:05 DEBUG "data": { 22:46:05 DEBUG "resultType": "vector", 22:46:05 DEBUG "result": [ 22:46:05 DEBUG { 22:46:05 DEBUG "metric": { 22:46:05 DEBUG "pod": "ds-idrepo-2" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG [...] 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG "pod": "ds-idrepo-0" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 1686692765, 22:46:05 DEBUG "583.3" 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG } 22:46:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692765" 22:46:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692765" 22:46:05 INFO [http_cmd]: http status code OK 22:46:05 DEBUG --- status code --- 22:46:05 DEBUG http status code is 200 (expected 200) 22:46:05 DEBUG --- http response --- 22:46:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:05 DEBUG ----- output ----- 22:46:05 DEBUG { 22:46:05 DEBUG "status": "success", 22:46:05 DEBUG "data": { 22:46:05 DEBUG "resultType": "vector", 22:46:05 DEBUG "result": [ 22:46:05 DEBUG { 22:46:05 DEBUG "metric": { 22:46:05 DEBUG "instance": "10.142.0.100:9100" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG [...] 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG "instance": "10.142.0.99:9100" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 1686692765, 22:46:05 DEBUG "19095650.066666666" 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG } 22:46:05 INFO [http_cmd]: http status code OK 22:46:05 DEBUG --- status code --- 22:46:05 DEBUG http status code is 200 (expected 200) 22:46:05 DEBUG --- http response --- 22:46:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:05 DEBUG ----- output ----- 22:46:05 DEBUG { 22:46:05 DEBUG "status": "success", 22:46:05 DEBUG "data": { 22:46:05 DEBUG "resultType": "vector", 22:46:05 DEBUG "result": [ 22:46:05 DEBUG { 22:46:05 DEBUG "metric": { 22:46:05 DEBUG "instance": "10.142.0.100:9100" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG [...] 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG "instance": "10.142.0.99:9100" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 1686692765, 22:46:05 DEBUG "19099838.5" 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG } 22:46:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692765" 22:46:05 INFO [http_cmd]: http status code OK 22:46:05 DEBUG --- status code --- 22:46:05 DEBUG http status code is 200 (expected 200) 22:46:05 DEBUG --- http response --- 22:46:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:05 DEBUG ----- output ----- 22:46:05 DEBUG { 22:46:05 DEBUG "status": "success", 22:46:05 DEBUG "data": { 22:46:05 DEBUG "resultType": "vector", 22:46:05 DEBUG "result": [ 22:46:05 DEBUG { 22:46:05 DEBUG "metric": { 22:46:05 DEBUG "pod": "ds-idrepo-2" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG [...] 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG "pod": "ds-idrepo-0" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 1686692765, 22:46:05 DEBUG "583.3" 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG } 22:46:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692765" 22:46:05 INFO [http_cmd]: http status code OK 22:46:05 DEBUG --- status code --- 22:46:05 DEBUG http status code is 200 (expected 200) 22:46:05 DEBUG --- http response --- 22:46:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:46:05 DEBUG ----- output ----- 22:46:05 DEBUG { 22:46:05 DEBUG "status": "success", 22:46:05 DEBUG "data": { 22:46:05 DEBUG "resultType": "vector", 22:46:05 DEBUG "result": [ 22:46:05 DEBUG { 22:46:05 DEBUG "metric": { 22:46:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG [...] 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG "pod": "idm-58dc667486-rgz9j" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 1686692765, 22:46:05 DEBUG "0" 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG } 22:46:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692765" 22:46:05 INFO [http_cmd]: http status code OK 22:46:05 DEBUG --- status code --- 22:46:05 DEBUG http status code is 200 (expected 200) 22:46:05 DEBUG --- http response --- 22:46:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:05 DEBUG ----- output ----- 22:46:05 DEBUG { 22:46:05 DEBUG "status": "success", 22:46:05 DEBUG "data": { 22:46:05 DEBUG "resultType": "vector", 22:46:05 DEBUG "result": [ 22:46:05 DEBUG { 22:46:05 DEBUG "metric": { 22:46:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG [...] 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG 22:46:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:46:05 DEBUG }, 22:46:05 DEBUG "value": [ 22:46:05 DEBUG 1686692765, 22:46:05 DEBUG "0.2406979425453452" 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG ] 22:46:05 DEBUG } 22:46:05 DEBUG } 22:46:23 INFO 22:46:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:46:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:46:23 INFO [loop_until]: OK (rc = 0) 22:46:23 DEBUG --- stdout --- 22:46:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3059m 13806Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2840m 4967Mi idm-58dc667486-rgz9j 2283m 5100Mi lodemon-6455d5ff7c-zzndq 3m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 197m 643Mi 22:46:23 DEBUG --- stderr --- 22:46:23 DEBUG 22:46:26 INFO 22:46:26 INFO [loop_until]: kubectl --namespace=xlou top node 22:46:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:46:26 INFO [loop_until]: OK (rc = 0) 22:46:26 DEBUG --- stdout --- 22:46:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2968m 18% 5906Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6867Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 73m 0% 6827Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2582m 16% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 424m 2% 3131Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1475Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 59m 0% 1171Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1200Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 72m 0% 13726Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3254m 20% 14513Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14606Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 274m 1% 1960Mi 3% 22:46:26 DEBUG --- stderr --- 22:46:26 DEBUG 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 DEBUG --- status code --- 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:04 DEBUG { 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG { 22:47:04 DEBUG { 22:47:04 DEBUG "data": { 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "data": { 22:47:04 DEBUG "data": { 22:47:04 DEBUG "result": [ 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG { 22:47:04 DEBUG "result": [ 22:47:04 DEBUG "result": [ 22:47:04 DEBUG "metric": { 22:47:04 DEBUG { 22:47:04 DEBUG { 22:47:04 DEBUG "pod": "ds-idrepo-2" 22:47:04 DEBUG "metric": { 22:47:04 DEBUG "metric": { 22:47:04 DEBUG }, 22:47:04 DEBUG "pod": "ds-cts-2" 22:47:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:47:04 DEBUG "value": [ 22:47:04 DEBUG }, 22:47:04 DEBUG }, 22:47:04 DEBUG 22:47:04 DEBUG "value": [ 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG [...] 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG [...] 22:47:04 DEBUG [...] 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG "pod": "ds-idrepo-0" 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG }, 22:47:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:47:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:47:04 DEBUG "value": [ 22:47:04 DEBUG }, 22:47:04 DEBUG }, 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG "value": [ 22:47:04 DEBUG "608.3" 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG ] 22:47:04 DEBUG "0" 22:47:04 DEBUG "0.03333333333333333" 22:47:04 DEBUG } 22:47:04 DEBUG ] 22:47:04 DEBUG ] 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG ] 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG { 22:47:04 DEBUG { 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "data": { 22:47:04 DEBUG "data": { 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "result": [ 22:47:04 DEBUG "result": [ 22:47:04 DEBUG { 22:47:04 DEBUG { 22:47:04 DEBUG "metric": { 22:47:04 DEBUG "metric": { 22:47:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:47:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:47:04 DEBUG }, 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG [...] 22:47:04 DEBUG [...] 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:47:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:47:04 DEBUG }, 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG "0" 22:47:04 DEBUG "0.2" 22:47:04 DEBUG ] 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG ] 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG { 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "data": { 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "result": [ 22:47:04 DEBUG { 22:47:04 DEBUG "metric": { 22:47:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG [...] 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG "0.03333333333333333" 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG { 22:47:04 DEBUG { 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "data": { 22:47:04 DEBUG "data": { 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "result": [ 22:47:04 DEBUG "result": [ 22:47:04 DEBUG { 22:47:04 DEBUG { 22:47:04 DEBUG "metric": { 22:47:04 DEBUG "metric": { 22:47:04 DEBUG "pod": "ds-cts-0" 22:47:04 DEBUG "pod": "ds-cts-2" 22:47:04 DEBUG }, 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 DEBUG [...] 22:47:04 DEBUG [...] 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (80 lines): 22:47:04 DEBUG "pod": "ds-idrepo-0" 22:47:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG }, 22:47:04 DEBUG }, 22:47:04 DEBUG { 22:47:04 DEBUG "value": [ 22:47:04 DEBUG "value": [ 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG "data": { 22:47:04 DEBUG "608.3" 22:47:04 DEBUG "0" 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG ] 22:47:04 DEBUG ] 22:47:04 DEBUG "result": [ 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG { 22:47:04 DEBUG ] 22:47:04 DEBUG ] 22:47:04 DEBUG "metric": { 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG "pod": "ds-cts-2" 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG [...] 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG "pod": "ds-idrepo-0" 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG "6807.12466531705" 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG { 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "data": { 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "result": [ 22:47:04 DEBUG { 22:47:04 DEBUG "metric": { 22:47:04 DEBUG "pod": "ds-cts-0" 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG [...] 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG "pod": "ds-idrepo-0" 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG "0" 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692824" 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG { 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "data": { 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "result": [ 22:47:04 DEBUG { 22:47:04 DEBUG "metric": { 22:47:04 DEBUG "instance": "10.142.0.100:9100" 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG [...] 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG "instance": "10.142.0.99:9100" 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG "0" 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692824" 22:47:04 INFO [http_cmd]: http status code OK 22:47:04 DEBUG --- status code --- 22:47:04 DEBUG http status code is 200 (expected 200) 22:47:04 DEBUG --- http response --- 22:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:47:04 DEBUG ----- output ----- 22:47:04 DEBUG { 22:47:04 DEBUG "status": "success", 22:47:04 DEBUG "data": { 22:47:04 DEBUG "resultType": "vector", 22:47:04 DEBUG "result": [ 22:47:04 DEBUG { 22:47:04 DEBUG "metric": { 22:47:04 DEBUG "pod": "ds-cts-0" 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG [...] 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG 22:47:04 DEBUG "pod": "ds-cts-2" 22:47:04 DEBUG }, 22:47:04 DEBUG "value": [ 22:47:04 DEBUG 1686692824, 22:47:04 DEBUG "0" 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG ] 22:47:04 DEBUG } 22:47:04 DEBUG } 22:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692825" 22:47:05 INFO [http_cmd]: http status code OK 22:47:05 DEBUG --- status code --- 22:47:05 DEBUG http status code is 200 (expected 200) 22:47:05 DEBUG --- http response --- 22:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692825" 22:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:05 DEBUG ----- output ----- 22:47:05 DEBUG { 22:47:05 DEBUG "status": "success", 22:47:05 DEBUG "data": { 22:47:05 DEBUG "resultType": "vector", 22:47:05 DEBUG "result": [ 22:47:05 DEBUG { 22:47:05 DEBUG "metric": { 22:47:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG [...] 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 1686692825, 22:47:05 DEBUG "0" 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG } 22:47:05 INFO [http_cmd]: http status code OK 22:47:05 DEBUG --- status code --- 22:47:05 DEBUG http status code is 200 (expected 200) 22:47:05 DEBUG --- http response --- 22:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:05 DEBUG ----- output ----- 22:47:05 DEBUG { 22:47:05 DEBUG "status": "success", 22:47:05 DEBUG "data": { 22:47:05 DEBUG "resultType": "vector", 22:47:05 DEBUG "result": [ 22:47:05 DEBUG { 22:47:05 DEBUG "metric": { 22:47:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG [...] 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 1686692825, 22:47:05 DEBUG "13616041984" 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG } 22:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692825" 22:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692825" 22:47:05 INFO [http_cmd]: http status code OK 22:47:05 DEBUG --- status code --- 22:47:05 DEBUG http status code is 200 (expected 200) 22:47:05 DEBUG --- http response --- 22:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:05 DEBUG ----- output ----- 22:47:05 DEBUG { 22:47:05 DEBUG "status": "success", 22:47:05 DEBUG "data": { 22:47:05 DEBUG "resultType": "vector", 22:47:05 DEBUG "result": [ 22:47:05 DEBUG { 22:47:05 DEBUG "metric": { 22:47:05 DEBUG "instance": "10.142.0.100:9100" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG [...] 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG "instance": "10.142.0.99:9100" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 1686692825, 22:47:05 DEBUG "0.00050000000000144" 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG } 22:47:05 INFO [http_cmd]: http status code OK 22:47:05 DEBUG --- status code --- 22:47:05 DEBUG http status code is 200 (expected 200) 22:47:05 DEBUG --- http response --- 22:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:05 DEBUG ----- output ----- 22:47:05 DEBUG { 22:47:05 DEBUG "status": "success", 22:47:05 DEBUG "data": { 22:47:05 DEBUG "resultType": "vector", 22:47:05 DEBUG "result": [ 22:47:05 DEBUG { 22:47:05 DEBUG "metric": { 22:47:05 DEBUG "pod": "ds-idrepo-2" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG [...] 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG "pod": "ds-idrepo-0" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 1686692825, 22:47:05 DEBUG "608.3" 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG } 22:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692825" 22:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692825" 22:47:05 INFO [http_cmd]: http status code OK 22:47:05 DEBUG --- status code --- 22:47:05 DEBUG http status code is 200 (expected 200) 22:47:05 DEBUG --- http response --- 22:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:05 DEBUG ----- output ----- 22:47:05 DEBUG { 22:47:05 DEBUG "status": "success", 22:47:05 DEBUG "data": { 22:47:05 DEBUG "resultType": "vector", 22:47:05 DEBUG "result": [ 22:47:05 DEBUG { 22:47:05 DEBUG "metric": { 22:47:05 DEBUG "instance": "10.142.0.100:9100" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG [...] 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG "instance": "10.142.0.99:9100" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 1686692825, 22:47:05 DEBUG "19560609.566666666" 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG } 22:47:05 INFO [http_cmd]: http status code OK 22:47:05 DEBUG --- status code --- 22:47:05 DEBUG http status code is 200 (expected 200) 22:47:05 DEBUG --- http response --- 22:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:05 DEBUG ----- output ----- 22:47:05 DEBUG { 22:47:05 DEBUG "status": "success", 22:47:05 DEBUG "data": { 22:47:05 DEBUG "resultType": "vector", 22:47:05 DEBUG "result": [ 22:47:05 DEBUG { 22:47:05 DEBUG "metric": { 22:47:05 DEBUG "instance": "10.142.0.100:9100" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG [...] 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG "instance": "10.142.0.99:9100" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 1686692825, 22:47:05 DEBUG "19564705.3" 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG } 22:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692825" 22:47:05 INFO [http_cmd]: http status code OK 22:47:05 DEBUG --- status code --- 22:47:05 DEBUG http status code is 200 (expected 200) 22:47:05 DEBUG --- http response --- 22:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:05 DEBUG ----- output ----- 22:47:05 DEBUG { 22:47:05 DEBUG "status": "success", 22:47:05 DEBUG "data": { 22:47:05 DEBUG "resultType": "vector", 22:47:05 DEBUG "result": [ 22:47:05 DEBUG { 22:47:05 DEBUG "metric": { 22:47:05 DEBUG "pod": "ds-idrepo-2" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG [...] 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG "pod": "ds-idrepo-0" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 1686692825, 22:47:05 DEBUG "608.3" 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG } 22:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692825" 22:47:05 INFO [http_cmd]: http status code OK 22:47:05 DEBUG --- status code --- 22:47:05 DEBUG http status code is 200 (expected 200) 22:47:05 DEBUG --- http response --- 22:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 22:47:05 DEBUG ----- output ----- 22:47:05 DEBUG { 22:47:05 DEBUG "status": "success", 22:47:05 DEBUG "data": { 22:47:05 DEBUG "resultType": "vector", 22:47:05 DEBUG "result": [ 22:47:05 DEBUG { 22:47:05 DEBUG "metric": { 22:47:05 DEBUG "pod": "ds-cts-2" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG [...] 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG "pod": "ds-idrepo-0" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 1686692825, 22:47:05 DEBUG "0.5550839156272448" 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG } 22:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692825" 22:47:05 INFO [http_cmd]: http status code OK 22:47:05 DEBUG --- status code --- 22:47:05 DEBUG http status code is 200 (expected 200) 22:47:05 DEBUG --- http response --- 22:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:05 DEBUG ----- output ----- 22:47:05 DEBUG { 22:47:05 DEBUG "status": "success", 22:47:05 DEBUG "data": { 22:47:05 DEBUG "resultType": "vector", 22:47:05 DEBUG "result": [ 22:47:05 DEBUG { 22:47:05 DEBUG "metric": { 22:47:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG [...] 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG 22:47:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:47:05 DEBUG }, 22:47:05 DEBUG "value": [ 22:47:05 DEBUG 1686692825, 22:47:05 DEBUG "0.2319375276480565" 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG ] 22:47:05 DEBUG } 22:47:05 DEBUG } 22:47:23 INFO 22:47:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:47:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:47:23 INFO [loop_until]: OK (rc = 0) 22:47:23 DEBUG --- stdout --- 22:47:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 10m 5712Mi am-7849cf7bdb-xrngh 8m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3369m 13822Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3039m 4968Mi idm-58dc667486-rgz9j 2628m 5100Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 213m 643Mi 22:47:23 DEBUG --- stderr --- 22:47:23 DEBUG 22:47:26 INFO 22:47:26 INFO [loop_until]: kubectl --namespace=xlou top node 22:47:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:47:26 INFO [loop_until]: OK (rc = 0) 22:47:26 DEBUG --- stdout --- 22:47:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3195m 20% 5909Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6867Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 71m 0% 6830Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2731m 17% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 449m 2% 3141Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1203Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13726Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3348m 21% 14516Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14608Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 280m 1% 1961Mi 3% 22:47:26 DEBUG --- stderr --- 22:47:26 DEBUG 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG { 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "data": { 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "result": [ 22:48:04 DEBUG { 22:48:04 DEBUG "metric": { 22:48:04 DEBUG "pod": "ds-cts-2" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG [...] 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG "0.000033977457103885745" 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG { 22:48:04 DEBUG { 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "data": { 22:48:04 DEBUG "data": { 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "result": [ 22:48:04 DEBUG "result": [ 22:48:04 DEBUG { 22:48:04 DEBUG { 22:48:04 DEBUG "metric": { 22:48:04 DEBUG "metric": { 22:48:04 DEBUG "pod": "ds-idrepo-2" 22:48:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:48:04 DEBUG }, 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG [...] 22:48:04 DEBUG [...] 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "pod": "ds-idrepo-0" 22:48:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:48:04 DEBUG }, 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG "635.6666666666666" 22:48:04 DEBUG "0" 22:48:04 DEBUG ] 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 DEBUG ] 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG { 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "data": { 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "result": [ 22:48:04 DEBUG { 22:48:04 DEBUG "metric": { 22:48:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG [...] 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG "0" 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG { 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "data": { 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "result": [ 22:48:04 DEBUG { 22:48:04 DEBUG "metric": { 22:48:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG [...] 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG "0.2" 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG { 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "data": { 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "result": [ 22:48:04 DEBUG { 22:48:04 DEBUG "metric": { 22:48:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG [...] 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG "0" 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG --- status code --- 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:48:04 DEBUG { 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG { 22:48:04 DEBUG "data": { 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "data": { 22:48:04 DEBUG "result": [ 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG { 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG "result": [ 22:48:04 DEBUG "metric": { 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG { 22:48:04 DEBUG "pod": "ds-cts-0" 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG "metric": { 22:48:04 DEBUG }, 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:48:04 DEBUG "pod": "ds-cts-2" 22:48:04 DEBUG "value": [ 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG }, 22:48:04 DEBUG 22:48:04 DEBUG { 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "data": { 22:48:04 DEBUG 22:48:04 DEBUG [...] 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "result": [ 22:48:04 DEBUG [...] 22:48:04 DEBUG 22:48:04 DEBUG { 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "metric": { 22:48:04 DEBUG 22:48:04 DEBUG "pod": "ds-idrepo-0" 22:48:04 DEBUG "pod": "ds-cts-2" 22:48:04 DEBUG 22:48:04 DEBUG }, 22:48:04 DEBUG }, 22:48:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:48:04 DEBUG "value": [ 22:48:04 DEBUG "value": [ 22:48:04 DEBUG }, 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG 22:48:04 DEBUG "value": [ 22:48:04 DEBUG "635.7" 22:48:04 DEBUG 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG ] 22:48:04 DEBUG 22:48:04 DEBUG "0" 22:48:04 DEBUG } 22:48:04 DEBUG [...] 22:48:04 DEBUG ] 22:48:04 DEBUG ] 22:48:04 DEBUG 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 DEBUG 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG 22:48:04 DEBUG } 22:48:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:48:04 DEBUG } 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG "0" 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG { 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "data": { 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "result": [ 22:48:04 DEBUG { 22:48:04 DEBUG "metric": { 22:48:04 DEBUG "pod": "ds-cts-0" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG [...] 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "pod": "ds-idrepo-0" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG "0" 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692884" 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG { 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "data": { 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "result": [ 22:48:04 DEBUG { 22:48:04 DEBUG "metric": { 22:48:04 DEBUG "instance": "10.142.0.100:9100" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG [...] 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "instance": "10.142.0.99:9100" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG "0" 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692884" 22:48:04 INFO [http_cmd]: http status code OK 22:48:04 DEBUG --- status code --- 22:48:04 DEBUG http status code is 200 (expected 200) 22:48:04 DEBUG --- http response --- 22:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:48:04 DEBUG ----- output ----- 22:48:04 DEBUG { 22:48:04 DEBUG "status": "success", 22:48:04 DEBUG "data": { 22:48:04 DEBUG "resultType": "vector", 22:48:04 DEBUG "result": [ 22:48:04 DEBUG { 22:48:04 DEBUG "metric": { 22:48:04 DEBUG "pod": "ds-cts-0" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG [...] 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG 22:48:04 DEBUG "pod": "ds-cts-2" 22:48:04 DEBUG }, 22:48:04 DEBUG "value": [ 22:48:04 DEBUG 1686692884, 22:48:04 DEBUG "0" 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG ] 22:48:04 DEBUG } 22:48:04 DEBUG } 22:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692885" 22:48:05 INFO [http_cmd]: http status code OK 22:48:05 DEBUG --- status code --- 22:48:05 DEBUG http status code is 200 (expected 200) 22:48:05 DEBUG --- http response --- 22:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:05 DEBUG ----- output ----- 22:48:05 DEBUG { 22:48:05 DEBUG "status": "success", 22:48:05 DEBUG "data": { 22:48:05 DEBUG "resultType": "vector", 22:48:05 DEBUG "result": [ 22:48:05 DEBUG { 22:48:05 DEBUG "metric": { 22:48:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG [...] 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 1686692885, 22:48:05 DEBUG "0" 22:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692885" 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG } 22:48:05 INFO [http_cmd]: http status code OK 22:48:05 DEBUG --- status code --- 22:48:05 DEBUG http status code is 200 (expected 200) 22:48:05 DEBUG --- http response --- 22:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:05 DEBUG ----- output ----- 22:48:05 DEBUG { 22:48:05 DEBUG "status": "success", 22:48:05 DEBUG "data": { 22:48:05 DEBUG "resultType": "vector", 22:48:05 DEBUG "result": [ 22:48:05 DEBUG { 22:48:05 DEBUG "metric": { 22:48:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG [...] 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 1686692885, 22:48:05 DEBUG "13615484928" 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG } 22:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692885" 22:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692885" 22:48:05 INFO [http_cmd]: http status code OK 22:48:05 DEBUG --- status code --- 22:48:05 DEBUG http status code is 200 (expected 200) 22:48:05 DEBUG --- http response --- 22:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:05 DEBUG ----- output ----- 22:48:05 DEBUG { 22:48:05 DEBUG "status": "success", 22:48:05 DEBUG "data": { 22:48:05 DEBUG "resultType": "vector", 22:48:05 DEBUG "result": [ 22:48:05 DEBUG { 22:48:05 DEBUG "metric": { 22:48:05 DEBUG "instance": "10.142.0.100:9100" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG [...] 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG "instance": "10.142.0.99:9100" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 1686692885, 22:48:05 DEBUG "0.00026666666666793997" 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG } 22:48:05 INFO [http_cmd]: http status code OK 22:48:05 DEBUG --- status code --- 22:48:05 DEBUG http status code is 200 (expected 200) 22:48:05 DEBUG --- http response --- 22:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:05 DEBUG ----- output ----- 22:48:05 DEBUG { 22:48:05 DEBUG "status": "success", 22:48:05 DEBUG "data": { 22:48:05 DEBUG "resultType": "vector", 22:48:05 DEBUG "result": [ 22:48:05 DEBUG { 22:48:05 DEBUG "metric": { 22:48:05 DEBUG "pod": "ds-idrepo-2" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG [...] 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG "pod": "ds-idrepo-0" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 1686692885, 22:48:05 DEBUG "635.6" 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG } 22:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692885" 22:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692885" 22:48:05 INFO [http_cmd]: http status code OK 22:48:05 DEBUG --- status code --- 22:48:05 DEBUG http status code is 200 (expected 200) 22:48:05 DEBUG --- http response --- 22:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:05 DEBUG ----- output ----- 22:48:05 DEBUG { 22:48:05 DEBUG "status": "success", 22:48:05 DEBUG "data": { 22:48:05 DEBUG "resultType": "vector", 22:48:05 DEBUG "result": [ 22:48:05 DEBUG { 22:48:05 DEBUG "metric": { 22:48:05 DEBUG "instance": "10.142.0.100:9100" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG [...] 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG "instance": "10.142.0.99:9100" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 1686692885, 22:48:05 DEBUG "19896224" 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG } 22:48:05 INFO [http_cmd]: http status code OK 22:48:05 DEBUG --- status code --- 22:48:05 DEBUG http status code is 200 (expected 200) 22:48:05 DEBUG --- http response --- 22:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:05 DEBUG ----- output ----- 22:48:05 DEBUG { 22:48:05 DEBUG "status": "success", 22:48:05 DEBUG "data": { 22:48:05 DEBUG "resultType": "vector", 22:48:05 DEBUG "result": [ 22:48:05 DEBUG { 22:48:05 DEBUG "metric": { 22:48:05 DEBUG "instance": "10.142.0.100:9100" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG [...] 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG "instance": "10.142.0.99:9100" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 1686692885, 22:48:05 DEBUG "19900455.666666664" 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG } 22:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692885" 22:48:05 INFO [http_cmd]: http status code OK 22:48:05 DEBUG --- status code --- 22:48:05 DEBUG http status code is 200 (expected 200) 22:48:05 DEBUG --- http response --- 22:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:05 DEBUG ----- output ----- 22:48:05 DEBUG { 22:48:05 DEBUG "status": "success", 22:48:05 DEBUG "data": { 22:48:05 DEBUG "resultType": "vector", 22:48:05 DEBUG "result": [ 22:48:05 DEBUG { 22:48:05 DEBUG "metric": { 22:48:05 DEBUG "pod": "ds-idrepo-2" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG [...] 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG "pod": "ds-idrepo-0" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 1686692885, 22:48:05 DEBUG "635.5666666666666" 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG } 22:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692885" 22:48:05 INFO [http_cmd]: http status code OK 22:48:05 DEBUG --- status code --- 22:48:05 DEBUG http status code is 200 (expected 200) 22:48:05 DEBUG --- http response --- 22:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:48:05 DEBUG ----- output ----- 22:48:05 DEBUG { 22:48:05 DEBUG "status": "success", 22:48:05 DEBUG "data": { 22:48:05 DEBUG "resultType": "vector", 22:48:05 DEBUG "result": [ 22:48:05 DEBUG { 22:48:05 DEBUG "metric": { 22:48:05 DEBUG "pod": "ds-cts-2" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG [...] 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 1686692885, 22:48:05 DEBUG "0" 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG } 22:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692885" 22:48:05 INFO [http_cmd]: http status code OK 22:48:05 DEBUG --- status code --- 22:48:05 DEBUG http status code is 200 (expected 200) 22:48:05 DEBUG --- http response --- 22:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:05 DEBUG ----- output ----- 22:48:05 DEBUG { 22:48:05 DEBUG "status": "success", 22:48:05 DEBUG "data": { 22:48:05 DEBUG "resultType": "vector", 22:48:05 DEBUG "result": [ 22:48:05 DEBUG { 22:48:05 DEBUG "metric": { 22:48:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG [...] 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG 22:48:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:48:05 DEBUG }, 22:48:05 DEBUG "value": [ 22:48:05 DEBUG 1686692885, 22:48:05 DEBUG "0.23792372433539422" 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG ] 22:48:05 DEBUG } 22:48:05 DEBUG } 22:48:23 INFO 22:48:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:48:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:48:23 INFO [loop_until]: OK (rc = 0) 22:48:23 DEBUG --- stdout --- 22:48:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 8m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3471m 13822Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 13m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3296m 4969Mi idm-58dc667486-rgz9j 2681m 5099Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 208m 643Mi 22:48:23 DEBUG --- stderr --- 22:48:23 DEBUG 22:48:27 INFO 22:48:27 INFO [loop_until]: kubectl --namespace=xlou top node 22:48:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:48:27 INFO [loop_until]: OK (rc = 0) 22:48:27 DEBUG --- stdout --- 22:48:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3242m 20% 5911Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6864Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6832Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2910m 18% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 436m 2% 3141Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 75m 0% 1475Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1203Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13727Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3617m 22% 14530Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 69m 0% 14609Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 274m 1% 1973Mi 3% 22:48:27 DEBUG --- stderr --- 22:48:27 DEBUG 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG --- status code --- 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:04 DEBUG { 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG { 22:49:04 DEBUG { 22:49:04 DEBUG "data": { 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "data": { 22:49:04 DEBUG "data": { 22:49:04 DEBUG "result": [ 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG { 22:49:04 DEBUG "result": [ 22:49:04 DEBUG "result": [ 22:49:04 DEBUG "metric": { 22:49:04 DEBUG { 22:49:04 DEBUG { 22:49:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:49:04 DEBUG "metric": { 22:49:04 DEBUG "metric": { 22:49:04 DEBUG }, 22:49:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:49:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:49:04 DEBUG "value": [ 22:49:04 DEBUG }, 22:49:04 DEBUG }, 22:49:04 DEBUG 22:49:04 DEBUG "value": [ 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG }, 22:49:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:49:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:49:04 DEBUG "value": [ 22:49:04 DEBUG }, 22:49:04 DEBUG }, 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG "value": [ 22:49:04 DEBUG "0.2" 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG ] 22:49:04 DEBUG "0" 22:49:04 DEBUG "0" 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG ] 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG { 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "data": { 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "result": [ 22:49:04 DEBUG { 22:49:04 DEBUG "metric": { 22:49:04 DEBUG "pod": "ds-idrepo-2" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG "pod": "ds-idrepo-0" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG "627.2333333333333" 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG { 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "data": { 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "result": [ 22:49:04 DEBUG { 22:49:04 DEBUG "metric": { 22:49:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG "0" 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG { 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "data": { 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "result": [ 22:49:04 DEBUG { 22:49:04 DEBUG "metric": { 22:49:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG "0" 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG { 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "data": { 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "result": [ 22:49:04 DEBUG { 22:49:04 DEBUG "metric": { 22:49:04 DEBUG "pod": "ds-cts-0" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG "pod": "ds-idrepo-0" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG "627.2333333333333" 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG { 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "data": { 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "result": [ 22:49:04 DEBUG { 22:49:04 DEBUG "metric": { 22:49:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG "0" 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG { 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "data": { 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "result": [ 22:49:04 DEBUG { 22:49:04 DEBUG "metric": { 22:49:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG "0" 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG { 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "data": { 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "result": [ 22:49:04 DEBUG { 22:49:04 DEBUG "metric": { 22:49:04 DEBUG "pod": "ds-cts-0" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG "pod": "ds-idrepo-0" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG "0" 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692944" 22:49:04 INFO [http_cmd]: http status code OK 22:49:04 DEBUG --- status code --- 22:49:04 DEBUG http status code is 200 (expected 200) 22:49:04 DEBUG --- http response --- 22:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:04 DEBUG ----- output ----- 22:49:04 DEBUG { 22:49:04 DEBUG "status": "success", 22:49:04 DEBUG "data": { 22:49:04 DEBUG "resultType": "vector", 22:49:04 DEBUG "result": [ 22:49:04 DEBUG { 22:49:04 DEBUG "metric": { 22:49:04 DEBUG "instance": "10.142.0.100:9100" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG [...] 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG 22:49:04 DEBUG "instance": "10.142.0.99:9100" 22:49:04 DEBUG }, 22:49:04 DEBUG "value": [ 22:49:04 DEBUG 1686692944, 22:49:04 DEBUG "0" 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG ] 22:49:04 DEBUG } 22:49:04 DEBUG } 22:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692944" 22:49:05 INFO [http_cmd]: http status code OK 22:49:05 DEBUG --- status code --- 22:49:05 DEBUG http status code is 200 (expected 200) 22:49:05 DEBUG --- http response --- 22:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:49:05 DEBUG ----- output ----- 22:49:05 DEBUG { 22:49:05 DEBUG "status": "success", 22:49:05 DEBUG "data": { 22:49:05 DEBUG "resultType": "vector", 22:49:05 DEBUG "result": [ 22:49:05 DEBUG { 22:49:05 DEBUG "metric": { 22:49:05 DEBUG "pod": "ds-cts-0" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG [...] 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG "pod": "ds-cts-2" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 1686692944, 22:49:05 DEBUG "0" 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG } 22:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692945" 22:49:05 INFO [http_cmd]: http status code OK 22:49:05 DEBUG --- status code --- 22:49:05 DEBUG http status code is 200 (expected 200) 22:49:05 DEBUG --- http response --- 22:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:05 DEBUG ----- output ----- 22:49:05 DEBUG { 22:49:05 DEBUG "status": "success", 22:49:05 DEBUG "data": { 22:49:05 DEBUG "resultType": "vector", 22:49:05 DEBUG "result": [ 22:49:05 DEBUG { 22:49:05 DEBUG "metric": { 22:49:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG [...] 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 1686692945, 22:49:05 DEBUG "0" 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG } 22:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686692945" 22:49:05 INFO [http_cmd]: http status code OK 22:49:05 DEBUG --- status code --- 22:49:05 DEBUG http status code is 200 (expected 200) 22:49:05 DEBUG --- http response --- 22:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:05 DEBUG ----- output ----- 22:49:05 DEBUG { 22:49:05 DEBUG "status": "success", 22:49:05 DEBUG "data": { 22:49:05 DEBUG "resultType": "vector", 22:49:05 DEBUG "result": [ 22:49:05 DEBUG { 22:49:05 DEBUG "metric": { 22:49:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG [...] 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 1686692945, 22:49:05 DEBUG "13617188864" 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG } 22:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692945" 22:49:05 INFO [http_cmd]: http status code OK 22:49:05 DEBUG --- status code --- 22:49:05 DEBUG http status code is 200 (expected 200) 22:49:05 DEBUG --- http response --- 22:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:05 DEBUG ----- output ----- 22:49:05 DEBUG { 22:49:05 DEBUG "status": "success", 22:49:05 DEBUG "data": { 22:49:05 DEBUG "resultType": "vector", 22:49:05 DEBUG "result": [ 22:49:05 DEBUG { 22:49:05 DEBUG "metric": { 22:49:05 DEBUG "pod": "ds-idrepo-2" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG [...] 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG "pod": "ds-idrepo-0" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 1686692945, 22:49:05 DEBUG "627.2333333333333" 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG } 22:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692945" 22:49:05 INFO [http_cmd]: http status code OK 22:49:05 DEBUG --- status code --- 22:49:05 DEBUG http status code is 200 (expected 200) 22:49:05 DEBUG --- http response --- 22:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:05 DEBUG ----- output ----- 22:49:05 DEBUG { 22:49:05 DEBUG "status": "success", 22:49:05 DEBUG "data": { 22:49:05 DEBUG "resultType": "vector", 22:49:05 DEBUG "result": [ 22:49:05 DEBUG { 22:49:05 DEBUG "metric": { 22:49:05 DEBUG "instance": "10.142.0.100:9100" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG [...] 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG "instance": "10.142.0.99:9100" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 1686692945, 22:49:05 DEBUG "0.00026666666666793997" 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG } 22:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692945" 22:49:05 INFO [http_cmd]: http status code OK 22:49:05 DEBUG --- status code --- 22:49:05 DEBUG http status code is 200 (expected 200) 22:49:05 DEBUG --- http response --- 22:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:05 DEBUG ----- output ----- 22:49:05 DEBUG { 22:49:05 DEBUG "status": "success", 22:49:05 DEBUG "data": { 22:49:05 DEBUG "resultType": "vector", 22:49:05 DEBUG "result": [ 22:49:05 DEBUG { 22:49:05 DEBUG "metric": { 22:49:05 DEBUG "instance": "10.142.0.100:9100" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG [...] 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG "instance": "10.142.0.99:9100" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 1686692945, 22:49:05 DEBUG "19980294.4" 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG } 22:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686692945" 22:49:05 INFO [http_cmd]: http status code OK 22:49:05 DEBUG --- status code --- 22:49:05 DEBUG http status code is 200 (expected 200) 22:49:05 DEBUG --- http response --- 22:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:05 DEBUG ----- output ----- 22:49:05 DEBUG { 22:49:05 DEBUG "status": "success", 22:49:05 DEBUG "data": { 22:49:05 DEBUG "resultType": "vector", 22:49:05 DEBUG "result": [ 22:49:05 DEBUG { 22:49:05 DEBUG "metric": { 22:49:05 DEBUG "instance": "10.142.0.100:9100" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG [...] 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG "instance": "10.142.0.99:9100" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 1686692945, 22:49:05 DEBUG "19984502.133333333" 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG } 22:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692945" 22:49:05 INFO [http_cmd]: http status code OK 22:49:05 DEBUG --- status code --- 22:49:05 DEBUG http status code is 200 (expected 200) 22:49:05 DEBUG --- http response --- 22:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:05 DEBUG ----- output ----- 22:49:05 DEBUG { 22:49:05 DEBUG "status": "success", 22:49:05 DEBUG "data": { 22:49:05 DEBUG "resultType": "vector", 22:49:05 DEBUG "result": [ 22:49:05 DEBUG { 22:49:05 DEBUG "metric": { 22:49:05 DEBUG "pod": "ds-idrepo-2" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG [...] 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG "pod": "ds-idrepo-0" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 1686692945, 22:49:05 DEBUG "627.2" 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG } 22:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686692945" 22:49:05 INFO [http_cmd]: http status code OK 22:49:05 DEBUG --- status code --- 22:49:05 DEBUG http status code is 200 (expected 200) 22:49:05 DEBUG --- http response --- 22:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:49:05 DEBUG ----- output ----- 22:49:05 DEBUG { 22:49:05 DEBUG "status": "success", 22:49:05 DEBUG "data": { 22:49:05 DEBUG "resultType": "vector", 22:49:05 DEBUG "result": [ 22:49:05 DEBUG { 22:49:05 DEBUG "metric": { 22:49:05 DEBUG "pod": "ds-cts-2" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG [...] 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG 22:49:05 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 22:49:05 DEBUG }, 22:49:05 DEBUG "value": [ 22:49:05 DEBUG 1686692945, 22:49:05 DEBUG "0" 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG ] 22:49:05 DEBUG } 22:49:05 DEBUG } 22:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686692945" 22:49:06 INFO [http_cmd]: http status code OK 22:49:06 DEBUG --- status code --- 22:49:06 DEBUG http status code is 200 (expected 200) 22:49:06 DEBUG --- http response --- 22:49:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:06 DEBUG ----- output ----- 22:49:06 DEBUG { 22:49:06 DEBUG "status": "success", 22:49:06 DEBUG "data": { 22:49:06 DEBUG "resultType": "vector", 22:49:06 DEBUG "result": [ 22:49:06 DEBUG { 22:49:06 DEBUG "metric": { 22:49:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:49:06 DEBUG }, 22:49:06 DEBUG "value": [ 22:49:06 DEBUG 22:49:06 DEBUG 22:49:06 DEBUG 22:49:06 DEBUG [...] 22:49:06 DEBUG 22:49:06 DEBUG 22:49:06 DEBUG 22:49:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:49:06 DEBUG }, 22:49:06 DEBUG "value": [ 22:49:06 DEBUG 1686692945, 22:49:06 DEBUG "0.23932637502654117" 22:49:06 DEBUG ] 22:49:06 DEBUG } 22:49:06 DEBUG ] 22:49:06 DEBUG } 22:49:06 DEBUG } 22:49:23 INFO 22:49:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:49:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:49:23 INFO [loop_until]: OK (rc = 0) 22:49:23 DEBUG --- stdout --- 22:49:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 10m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 8m 417Mi ds-cts-1 9m 425Mi ds-cts-2 10m 393Mi ds-idrepo-0 3487m 13806Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3206m 4972Mi idm-58dc667486-rgz9j 2566m 5099Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 198m 643Mi 22:49:23 DEBUG --- stderr --- 22:49:23 DEBUG 22:49:27 INFO 22:49:27 INFO [loop_until]: kubectl --namespace=xlou top node 22:49:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:49:27 INFO [loop_until]: OK (rc = 0) 22:49:27 DEBUG --- stdout --- 22:49:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3328m 20% 5906Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6864Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 77m 0% 6831Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2659m 16% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 75m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 446m 2% 3144Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 83m 0% 1476Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13727Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3441m 21% 14515Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14610Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 265m 1% 1964Mi 3% 22:49:27 DEBUG --- stderr --- 22:49:27 DEBUG 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG { 22:50:04 DEBUG { 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "data": { 22:50:04 DEBUG "data": { 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "result": [ 22:50:04 DEBUG "result": [ 22:50:04 DEBUG { 22:50:04 DEBUG { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:50:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:50:04 DEBUG }, 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG [...] 22:50:04 DEBUG [...] 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:50:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:50:04 DEBUG }, 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG "0" 22:50:04 DEBUG "0.004123854903260612" 22:50:04 DEBUG ] 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 DEBUG ] 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG { 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "data": { 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "result": [ 22:50:04 DEBUG { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG [...] 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG "0.2" 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG { 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "data": { 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "result": [ 22:50:04 DEBUG { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "pod": "ds-idrepo-2" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG [...] 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG "pod": "ds-idrepo-0" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG "637.3" 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG { 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "data": { 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "result": [ 22:50:04 DEBUG { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG [...] 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG "0" 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG { 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "data": { 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "result": [ 22:50:04 DEBUG { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG [...] 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG "0" 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG { 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "data": { 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "result": [ 22:50:04 DEBUG { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG [...] 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG "152067057.44150826" 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG { 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "data": { 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "result": [ 22:50:04 DEBUG { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "pod": "ds-cts-0" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG [...] 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG "pod": "ds-idrepo-0" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG "637.2333333333333" 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG { 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "data": { 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "result": [ 22:50:04 DEBUG { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG [...] 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG "0" 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693004" 22:50:04 INFO [http_cmd]: http status code OK 22:50:04 DEBUG --- status code --- 22:50:04 DEBUG http status code is 200 (expected 200) 22:50:04 DEBUG --- http response --- 22:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:04 DEBUG ----- output ----- 22:50:04 DEBUG { 22:50:04 DEBUG "status": "success", 22:50:04 DEBUG "data": { 22:50:04 DEBUG "resultType": "vector", 22:50:04 DEBUG "result": [ 22:50:04 DEBUG { 22:50:04 DEBUG "metric": { 22:50:04 DEBUG "pod": "ds-cts-0" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG [...] 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG 22:50:04 DEBUG "pod": "ds-idrepo-0" 22:50:04 DEBUG }, 22:50:04 DEBUG "value": [ 22:50:04 DEBUG 1686693004, 22:50:04 DEBUG "0" 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG ] 22:50:04 DEBUG } 22:50:04 DEBUG } 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693005" 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "instance": "10.142.0.100:9100" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "instance": "10.142.0.99:9100" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "0" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693005" 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "pod": "ds-cts-0" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "pod": "ds-cts-2" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "0" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693005" 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693005" 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "13617528832" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "0" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693005" 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "pod": "ds-idrepo-2" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "pod": "ds-idrepo-0" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "637.3333333333334" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693005" 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "instance": "10.142.0.100:9100" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "instance": "10.142.0.99:9100" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "0.0002999999999985903" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693005" 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "instance": "10.142.0.100:9100" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "instance": "10.142.0.99:9100" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "20987276.5" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693005" 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "instance": "10.142.0.100:9100" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "instance": "10.142.0.99:9100" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "20991185.3" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693005" 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "pod": "ds-idrepo-2" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "pod": "ds-idrepo-0" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "637.3666666666667" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693005" 22:50:05 INFO [http_cmd]: http status code OK 22:50:05 DEBUG --- status code --- 22:50:05 DEBUG http status code is 200 (expected 200) 22:50:05 DEBUG --- http response --- 22:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:50:05 DEBUG ----- output ----- 22:50:05 DEBUG { 22:50:05 DEBUG "status": "success", 22:50:05 DEBUG "data": { 22:50:05 DEBUG "resultType": "vector", 22:50:05 DEBUG "result": [ 22:50:05 DEBUG { 22:50:05 DEBUG "metric": { 22:50:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG [...] 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG 22:50:05 DEBUG "pod": "idm-58dc667486-rgz9j" 22:50:05 DEBUG }, 22:50:05 DEBUG "value": [ 22:50:05 DEBUG 1686693005, 22:50:05 DEBUG "0" 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG ] 22:50:05 DEBUG } 22:50:05 DEBUG } 22:50:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693006" 22:50:06 INFO [http_cmd]: http status code OK 22:50:06 DEBUG --- status code --- 22:50:06 DEBUG http status code is 200 (expected 200) 22:50:06 DEBUG --- http response --- 22:50:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:06 DEBUG ----- output ----- 22:50:06 DEBUG { 22:50:06 DEBUG "status": "success", 22:50:06 DEBUG "data": { 22:50:06 DEBUG "resultType": "vector", 22:50:06 DEBUG "result": [ 22:50:06 DEBUG { 22:50:06 DEBUG "metric": { 22:50:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:50:06 DEBUG }, 22:50:06 DEBUG "value": [ 22:50:06 DEBUG 22:50:06 DEBUG 22:50:06 DEBUG 22:50:06 DEBUG [...] 22:50:06 DEBUG 22:50:06 DEBUG 22:50:06 DEBUG 22:50:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:50:06 DEBUG }, 22:50:06 DEBUG "value": [ 22:50:06 DEBUG 1686693006, 22:50:06 DEBUG "0.24005560205127302" 22:50:06 DEBUG ] 22:50:06 DEBUG } 22:50:06 DEBUG ] 22:50:06 DEBUG } 22:50:06 DEBUG } 22:50:23 INFO 22:50:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:50:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:50:23 INFO [loop_until]: OK (rc = 0) 22:50:23 DEBUG --- stdout --- 22:50:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 10m 5688Mi ds-cts-0 9m 418Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3568m 13823Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3412m 4979Mi idm-58dc667486-rgz9j 2754m 5102Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 206m 643Mi 22:50:23 DEBUG --- stderr --- 22:50:23 DEBUG 22:50:27 INFO 22:50:27 INFO [loop_until]: kubectl --namespace=xlou top node 22:50:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:50:27 INFO [loop_until]: OK (rc = 0) 22:50:27 DEBUG --- stdout --- 22:50:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3572m 22% 5915Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6861Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6832Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2886m 18% 6081Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 77m 0% 6961Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 437m 2% 3144Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 58m 0% 1163Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1200Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 72m 0% 13727Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3572m 22% 14513Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14609Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 275m 1% 1965Mi 3% 22:50:27 DEBUG --- stderr --- 22:50:27 DEBUG 22:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693064" 22:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693064" 22:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693064" 22:51:04 INFO [http_cmd]: http status code OK 22:51:04 INFO [http_cmd]: http status code OK 22:51:04 DEBUG --- status code --- 22:51:04 DEBUG --- status code --- 22:51:04 DEBUG http status code is 200 (expected 200) 22:51:04 DEBUG http status code is 200 (expected 200) 22:51:04 DEBUG --- http response --- 22:51:04 DEBUG --- http response --- 22:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:04 DEBUG ----- output ----- 22:51:04 DEBUG ----- output ----- 22:51:04 DEBUG { 22:51:04 DEBUG { 22:51:04 DEBUG "status": "success", 22:51:04 DEBUG "status": "success", 22:51:04 DEBUG "data": { 22:51:04 DEBUG "data": { 22:51:04 DEBUG "resultType": "vector", 22:51:04 DEBUG "resultType": "vector", 22:51:04 DEBUG "result": [ 22:51:04 DEBUG "result": [ 22:51:04 DEBUG { 22:51:04 DEBUG { 22:51:04 DEBUG "metric": { 22:51:04 DEBUG "metric": { 22:51:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:51:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:51:04 DEBUG }, 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG [...] 22:51:04 INFO [http_cmd]: http status code OK 22:51:04 DEBUG [...] 22:51:04 DEBUG 22:51:04 DEBUG --- status code --- 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG http status code is 200 (expected 200) 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG --- http response --- 22:51:04 DEBUG 22:51:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:51:04 DEBUG }, 22:51:04 DEBUG ----- output ----- 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG { 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 1686693064, 22:51:04 DEBUG "status": "success", 22:51:04 DEBUG 1686693064, 22:51:04 DEBUG "0.2" 22:51:04 DEBUG "data": { 22:51:04 DEBUG "0" 22:51:04 DEBUG ] 22:51:04 DEBUG "resultType": "vector", 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG "result": [ 22:51:04 DEBUG } 22:51:04 DEBUG ] 22:51:04 DEBUG { 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG "metric": { 22:51:04 DEBUG } 22:51:04 DEBUG } 22:51:04 DEBUG "pod": "ds-cts-2" 22:51:04 DEBUG } 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG [...] 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 1686693064, 22:51:04 DEBUG "0" 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG } 22:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693064" 22:51:04 INFO [http_cmd]: http status code OK 22:51:04 DEBUG --- status code --- 22:51:04 DEBUG http status code is 200 (expected 200) 22:51:04 DEBUG --- http response --- 22:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:04 DEBUG ----- output ----- 22:51:04 DEBUG { 22:51:04 DEBUG "status": "success", 22:51:04 DEBUG "data": { 22:51:04 DEBUG "resultType": "vector", 22:51:04 DEBUG "result": [ 22:51:04 DEBUG { 22:51:04 DEBUG "metric": { 22:51:04 DEBUG "pod": "ds-idrepo-2" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG [...] 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG "pod": "ds-idrepo-0" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 1686693064, 22:51:04 DEBUG "580.5333333333333" 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG } 22:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693064" 22:51:04 INFO [http_cmd]: http status code OK 22:51:04 DEBUG --- status code --- 22:51:04 DEBUG http status code is 200 (expected 200) 22:51:04 DEBUG --- http response --- 22:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:04 DEBUG ----- output ----- 22:51:04 DEBUG { 22:51:04 DEBUG "status": "success", 22:51:04 DEBUG "data": { 22:51:04 DEBUG "resultType": "vector", 22:51:04 DEBUG "result": [ 22:51:04 DEBUG { 22:51:04 DEBUG "metric": { 22:51:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG [...] 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 1686693064, 22:51:04 DEBUG "0" 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG } 22:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693064" 22:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693064" 22:51:04 INFO [http_cmd]: http status code OK 22:51:04 DEBUG --- status code --- 22:51:04 DEBUG http status code is 200 (expected 200) 22:51:04 DEBUG --- http response --- 22:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:04 DEBUG ----- output ----- 22:51:04 DEBUG { 22:51:04 DEBUG "status": "success", 22:51:04 DEBUG "data": { 22:51:04 DEBUG "resultType": "vector", 22:51:04 DEBUG "result": [ 22:51:04 DEBUG { 22:51:04 DEBUG "metric": { 22:51:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG [...] 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 1686693064, 22:51:04 DEBUG "0" 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG } 22:51:04 INFO [http_cmd]: http status code OK 22:51:04 DEBUG --- status code --- 22:51:04 DEBUG http status code is 200 (expected 200) 22:51:04 DEBUG --- http response --- 22:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:04 DEBUG ----- output ----- 22:51:04 DEBUG { 22:51:04 DEBUG "status": "success", 22:51:04 DEBUG "data": { 22:51:04 DEBUG "resultType": "vector", 22:51:04 DEBUG "result": [ 22:51:04 DEBUG { 22:51:04 DEBUG "metric": { 22:51:04 DEBUG "pod": "ds-cts-0" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG [...] 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG "pod": "ds-idrepo-0" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 1686693064, 22:51:04 DEBUG "580.5" 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG } 22:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693064" 22:51:04 INFO [http_cmd]: http status code OK 22:51:04 DEBUG --- status code --- 22:51:04 DEBUG http status code is 200 (expected 200) 22:51:04 DEBUG --- http response --- 22:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:51:04 DEBUG ----- output ----- 22:51:04 DEBUG { 22:51:04 DEBUG "status": "success", 22:51:04 DEBUG "data": { 22:51:04 DEBUG "resultType": "vector", 22:51:04 DEBUG "result": [ 22:51:04 DEBUG { 22:51:04 DEBUG "metric": { 22:51:04 DEBUG "pod": "ds-cts-2" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG [...] 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG "pod": "ds-idrepo-0" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 1686693064, 22:51:04 DEBUG "6415.145592473581" 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG } 22:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693064" 22:51:04 INFO [http_cmd]: http status code OK 22:51:04 DEBUG --- status code --- 22:51:04 DEBUG http status code is 200 (expected 200) 22:51:04 DEBUG --- http response --- 22:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:04 DEBUG ----- output ----- 22:51:04 DEBUG { 22:51:04 DEBUG "status": "success", 22:51:04 DEBUG "data": { 22:51:04 DEBUG "resultType": "vector", 22:51:04 DEBUG "result": [ 22:51:04 DEBUG { 22:51:04 DEBUG "metric": { 22:51:04 DEBUG "pod": "ds-cts-2" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG [...] 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG 22:51:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:51:04 DEBUG }, 22:51:04 DEBUG "value": [ 22:51:04 DEBUG 1686693064, 22:51:04 DEBUG "0" 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG ] 22:51:04 DEBUG } 22:51:04 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "pod": "ds-cts-0" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "pod": "ds-idrepo-0" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "0" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "instance": "10.142.0.100:9100" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "instance": "10.142.0.99:9100" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "0" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "pod": "ds-cts-0" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "pod": "ds-cts-2" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "0" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "0" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "13616664576" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "pod": "ds-idrepo-2" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "pod": "ds-idrepo-0" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "580.5333333333333" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "instance": "10.142.0.100:9100" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "instance": "10.142.0.99:9100" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "0.00026666666666793997" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "instance": "10.142.0.100:9100" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "instance": "10.142.0.99:9100" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "19341411.3" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "instance": "10.142.0.100:9100" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "instance": "10.142.0.99:9100" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "19345296" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693065" 22:51:05 INFO [http_cmd]: http status code OK 22:51:05 DEBUG --- status code --- 22:51:05 DEBUG http status code is 200 (expected 200) 22:51:05 DEBUG --- http response --- 22:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:05 DEBUG ----- output ----- 22:51:05 DEBUG { 22:51:05 DEBUG "status": "success", 22:51:05 DEBUG "data": { 22:51:05 DEBUG "resultType": "vector", 22:51:05 DEBUG "result": [ 22:51:05 DEBUG { 22:51:05 DEBUG "metric": { 22:51:05 DEBUG "pod": "ds-idrepo-2" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG [...] 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG 22:51:05 DEBUG "pod": "ds-idrepo-0" 22:51:05 DEBUG }, 22:51:05 DEBUG "value": [ 22:51:05 DEBUG 1686693065, 22:51:05 DEBUG "580.5333333333333" 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG ] 22:51:05 DEBUG } 22:51:05 DEBUG } 22:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693065" 22:51:06 INFO [http_cmd]: http status code OK 22:51:06 DEBUG --- status code --- 22:51:06 DEBUG http status code is 200 (expected 200) 22:51:06 DEBUG --- http response --- 22:51:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:51:06 DEBUG ----- output ----- 22:51:06 DEBUG { 22:51:06 DEBUG "status": "success", 22:51:06 DEBUG "data": { 22:51:06 DEBUG "resultType": "vector", 22:51:06 DEBUG "result": [ 22:51:06 DEBUG { 22:51:06 DEBUG "metric": { 22:51:06 DEBUG "pod": "ds-cts-2" 22:51:06 DEBUG }, 22:51:06 DEBUG "value": [ 22:51:06 DEBUG 22:51:06 DEBUG 22:51:06 DEBUG 22:51:06 DEBUG [...] 22:51:06 DEBUG 22:51:06 DEBUG 22:51:06 DEBUG 22:51:06 DEBUG "pod": "ds-idrepo-0" 22:51:06 DEBUG }, 22:51:06 DEBUG "value": [ 22:51:06 DEBUG 1686693065, 22:51:06 DEBUG "0.7287905389737304" 22:51:06 DEBUG ] 22:51:06 DEBUG } 22:51:06 DEBUG ] 22:51:06 DEBUG } 22:51:06 DEBUG } 22:51:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693066" 22:51:06 INFO [http_cmd]: http status code OK 22:51:06 DEBUG --- status code --- 22:51:06 DEBUG http status code is 200 (expected 200) 22:51:06 DEBUG --- http response --- 22:51:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:06 DEBUG ----- output ----- 22:51:06 DEBUG { 22:51:06 DEBUG "status": "success", 22:51:06 DEBUG "data": { 22:51:06 DEBUG "resultType": "vector", 22:51:06 DEBUG "result": [ 22:51:06 DEBUG { 22:51:06 DEBUG "metric": { 22:51:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:51:06 DEBUG }, 22:51:06 DEBUG "value": [ 22:51:06 DEBUG 22:51:06 DEBUG 22:51:06 DEBUG 22:51:06 DEBUG [...] 22:51:06 DEBUG 22:51:06 DEBUG 22:51:06 DEBUG 22:51:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:51:06 DEBUG }, 22:51:06 DEBUG "value": [ 22:51:06 DEBUG 1686693066, 22:51:06 DEBUG "0.22595965618261268" 22:51:06 DEBUG ] 22:51:06 DEBUG } 22:51:06 DEBUG ] 22:51:06 DEBUG } 22:51:06 DEBUG } 22:51:23 INFO 22:51:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:51:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:51:23 INFO [loop_until]: OK (rc = 0) 22:51:23 DEBUG --- stdout --- 22:51:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 8m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3196m 13799Mi ds-idrepo-1 15m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3085m 4981Mi idm-58dc667486-rgz9j 2671m 5101Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 213m 644Mi 22:51:23 DEBUG --- stderr --- 22:51:23 DEBUG 22:51:27 INFO 22:51:27 INFO [loop_until]: kubectl --namespace=xlou top node 22:51:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:51:27 INFO [loop_until]: OK (rc = 0) 22:51:27 DEBUG --- stdout --- 22:51:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3099m 19% 5918Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6865Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6832Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2881m 18% 6082Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 453m 2% 3125Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 72m 0% 13726Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3388m 21% 14513Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14610Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 267m 1% 1965Mi 3% 22:51:27 DEBUG --- stderr --- 22:51:27 DEBUG 22:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693124" 22:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693124" 22:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693124" 22:52:04 INFO [http_cmd]: http status code OK 22:52:04 INFO [http_cmd]: http status code OK 22:52:04 INFO [http_cmd]: http status code OK 22:52:04 DEBUG --- status code --- 22:52:04 DEBUG --- status code --- 22:52:04 DEBUG --- status code --- 22:52:04 DEBUG http status code is 200 (expected 200) 22:52:04 DEBUG http status code is 200 (expected 200) 22:52:04 DEBUG http status code is 200 (expected 200) 22:52:04 DEBUG --- http response --- 22:52:04 DEBUG --- http response --- 22:52:04 DEBUG --- http response --- 22:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:04 DEBUG ----- output ----- 22:52:04 DEBUG ----- output ----- 22:52:04 DEBUG ----- output ----- 22:52:04 DEBUG { 22:52:04 DEBUG { 22:52:04 DEBUG { 22:52:04 DEBUG "status": "success", 22:52:04 DEBUG "status": "success", 22:52:04 DEBUG "status": "success", 22:52:04 DEBUG "data": { 22:52:04 DEBUG "data": { 22:52:04 DEBUG "data": { 22:52:04 DEBUG "resultType": "vector", 22:52:04 DEBUG "resultType": "vector", 22:52:04 DEBUG "resultType": "vector", 22:52:04 DEBUG "result": [ 22:52:04 DEBUG "result": [ 22:52:04 DEBUG "result": [ 22:52:04 DEBUG { 22:52:04 DEBUG { 22:52:04 DEBUG { 22:52:04 DEBUG "metric": { 22:52:04 DEBUG "metric": { 22:52:04 DEBUG "metric": { 22:52:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:52:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:52:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:52:04 DEBUG }, 22:52:04 DEBUG }, 22:52:04 DEBUG }, 22:52:04 DEBUG "value": [ 22:52:04 DEBUG "value": [ 22:52:04 DEBUG "value": [ 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG [...] 22:52:04 DEBUG [...] 22:52:04 DEBUG [...] 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:52:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:52:04 DEBUG "pod": "idm-58dc667486-rgz9j" 22:52:04 DEBUG }, 22:52:04 DEBUG }, 22:52:04 DEBUG }, 22:52:04 DEBUG "value": [ 22:52:04 DEBUG "value": [ 22:52:04 DEBUG "value": [ 22:52:04 DEBUG 1686693124, 22:52:04 DEBUG 1686693124, 22:52:04 DEBUG 1686693124, 22:52:04 DEBUG "0.2" 22:52:04 DEBUG "0" 22:52:04 DEBUG "2.559288907898" 22:52:04 DEBUG ] 22:52:04 DEBUG ] 22:52:04 DEBUG ] 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:04 DEBUG ] 22:52:04 DEBUG ] 22:52:04 DEBUG ] 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693124" 22:52:04 INFO [http_cmd]: http status code OK 22:52:04 DEBUG --- status code --- 22:52:04 DEBUG http status code is 200 (expected 200) 22:52:04 DEBUG --- http response --- 22:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:04 DEBUG ----- output ----- 22:52:04 DEBUG { 22:52:04 DEBUG "status": "success", 22:52:04 DEBUG "data": { 22:52:04 DEBUG "resultType": "vector", 22:52:04 DEBUG "result": [ 22:52:04 DEBUG { 22:52:04 DEBUG "metric": { 22:52:04 DEBUG "pod": "ds-idrepo-2" 22:52:04 DEBUG }, 22:52:04 DEBUG "value": [ 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG [...] 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG "pod": "ds-idrepo-0" 22:52:04 DEBUG }, 22:52:04 DEBUG "value": [ 22:52:04 DEBUG 1686693124, 22:52:04 DEBUG "598.9333333333333" 22:52:04 DEBUG ] 22:52:04 DEBUG } 22:52:04 DEBUG ] 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693124" 22:52:04 INFO [http_cmd]: http status code OK 22:52:04 DEBUG --- status code --- 22:52:04 DEBUG http status code is 200 (expected 200) 22:52:04 DEBUG --- http response --- 22:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:04 DEBUG ----- output ----- 22:52:04 DEBUG { 22:52:04 DEBUG "status": "success", 22:52:04 DEBUG "data": { 22:52:04 DEBUG "resultType": "vector", 22:52:04 DEBUG "result": [ 22:52:04 DEBUG { 22:52:04 DEBUG "metric": { 22:52:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:52:04 DEBUG }, 22:52:04 DEBUG "value": [ 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG [...] 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:52:04 DEBUG }, 22:52:04 DEBUG "value": [ 22:52:04 DEBUG 1686693124, 22:52:04 DEBUG "0" 22:52:04 DEBUG ] 22:52:04 DEBUG } 22:52:04 DEBUG ] 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693124" 22:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693124" 22:52:04 INFO [http_cmd]: http status code OK 22:52:04 DEBUG --- status code --- 22:52:04 DEBUG http status code is 200 (expected 200) 22:52:04 DEBUG --- http response --- 22:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:04 DEBUG ----- output ----- 22:52:04 DEBUG { 22:52:04 DEBUG "status": "success", 22:52:04 DEBUG "data": { 22:52:04 DEBUG "resultType": "vector", 22:52:04 DEBUG "result": [ 22:52:04 DEBUG { 22:52:04 DEBUG "metric": { 22:52:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:52:04 DEBUG }, 22:52:04 DEBUG "value": [ 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG [...] 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG 22:52:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:52:04 DEBUG }, 22:52:04 DEBUG "value": [ 22:52:04 DEBUG 1686693124, 22:52:04 DEBUG "0" 22:52:04 DEBUG ] 22:52:04 DEBUG } 22:52:04 DEBUG ] 22:52:04 DEBUG } 22:52:04 DEBUG } 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "pod": "ds-cts-0" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "pod": "ds-idrepo-0" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693124, 22:52:05 DEBUG "598.9666666666667" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "pod": "idm-58dc667486-rgz9j" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "0" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "pod": "idm-58dc667486-rgz9j" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "200563791.05009803" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "pod": "ds-cts-0" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "pod": "ds-idrepo-0" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "0" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "instance": "10.142.0.100:9100" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "instance": "10.142.0.99:9100" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "0" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "pod": "ds-cts-0" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "pod": "ds-cts-2" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "0" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "0" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "13616836608" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "pod": "ds-idrepo-2" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "pod": "ds-idrepo-0" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "598.8666666666667" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "instance": "10.142.0.100:9100" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "instance": "10.142.0.99:9100" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "0.00023333333333350008" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "instance": "10.142.0.100:9100" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "instance": "10.142.0.99:9100" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "18612982.066666666" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "instance": "10.142.0.100:9100" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "instance": "10.142.0.99:9100" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "18615375.733333334" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693125" 22:52:05 INFO [http_cmd]: http status code OK 22:52:05 DEBUG --- status code --- 22:52:05 DEBUG http status code is 200 (expected 200) 22:52:05 DEBUG --- http response --- 22:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:05 DEBUG ----- output ----- 22:52:05 DEBUG { 22:52:05 DEBUG "status": "success", 22:52:05 DEBUG "data": { 22:52:05 DEBUG "resultType": "vector", 22:52:05 DEBUG "result": [ 22:52:05 DEBUG { 22:52:05 DEBUG "metric": { 22:52:05 DEBUG "pod": "ds-idrepo-2" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG [...] 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG 22:52:05 DEBUG "pod": "ds-idrepo-0" 22:52:05 DEBUG }, 22:52:05 DEBUG "value": [ 22:52:05 DEBUG 1686693125, 22:52:05 DEBUG "598.8666666666667" 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG ] 22:52:05 DEBUG } 22:52:05 DEBUG } 22:52:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693126" 22:52:06 INFO [http_cmd]: http status code OK 22:52:06 DEBUG --- status code --- 22:52:06 DEBUG http status code is 200 (expected 200) 22:52:06 DEBUG --- http response --- 22:52:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:06 DEBUG ----- output ----- 22:52:06 DEBUG { 22:52:06 DEBUG "status": "success", 22:52:06 DEBUG "data": { 22:52:06 DEBUG "resultType": "vector", 22:52:06 DEBUG "result": [ 22:52:06 DEBUG { 22:52:06 DEBUG "metric": { 22:52:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:52:06 DEBUG }, 22:52:06 DEBUG "value": [ 22:52:06 DEBUG 22:52:06 DEBUG 22:52:06 DEBUG 22:52:06 DEBUG [...] 22:52:06 DEBUG 22:52:06 DEBUG 22:52:06 DEBUG 22:52:06 DEBUG "pod": "idm-58dc667486-rgz9j" 22:52:06 DEBUG }, 22:52:06 DEBUG "value": [ 22:52:06 DEBUG 1686693126, 22:52:06 DEBUG "0" 22:52:06 DEBUG ] 22:52:06 DEBUG } 22:52:06 DEBUG ] 22:52:06 DEBUG } 22:52:06 DEBUG } 22:52:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693126" 22:52:06 INFO [http_cmd]: http status code OK 22:52:06 DEBUG --- status code --- 22:52:06 DEBUG http status code is 200 (expected 200) 22:52:06 DEBUG --- http response --- 22:52:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:06 DEBUG ----- output ----- 22:52:06 DEBUG { 22:52:06 DEBUG "status": "success", 22:52:06 DEBUG "data": { 22:52:06 DEBUG "resultType": "vector", 22:52:06 DEBUG "result": [ 22:52:06 DEBUG { 22:52:06 DEBUG "metric": { 22:52:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:52:06 DEBUG }, 22:52:06 DEBUG "value": [ 22:52:06 DEBUG 22:52:06 DEBUG 22:52:06 DEBUG 22:52:06 DEBUG [...] 22:52:06 DEBUG 22:52:06 DEBUG 22:52:06 DEBUG 22:52:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:52:06 DEBUG }, 22:52:06 DEBUG "value": [ 22:52:06 DEBUG 1686693126, 22:52:06 DEBUG "0.24182939288383212" 22:52:06 DEBUG ] 22:52:06 DEBUG } 22:52:06 DEBUG ] 22:52:06 DEBUG } 22:52:06 DEBUG } 22:52:23 INFO 22:52:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:52:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:52:23 INFO [loop_until]: OK (rc = 0) 22:52:23 DEBUG --- stdout --- 22:52:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 8m 417Mi ds-cts-1 8m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3212m 13800Mi ds-idrepo-1 15m 13821Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2928m 4983Mi idm-58dc667486-rgz9j 2756m 5101Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 210m 644Mi 22:52:23 DEBUG --- stderr --- 22:52:23 DEBUG 22:52:27 INFO 22:52:27 INFO [loop_until]: kubectl --namespace=xlou top node 22:52:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:52:27 INFO [loop_until]: OK (rc = 0) 22:52:27 DEBUG --- stdout --- 22:52:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2960m 18% 5920Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 69m 0% 6833Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2693m 16% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6946Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 432m 2% 3137Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1201Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13728Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3137m 19% 14515Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14609Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 272m 1% 1965Mi 3% 22:52:27 DEBUG --- stderr --- 22:52:27 DEBUG 22:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693184" 22:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693184" 22:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693184" 22:53:04 INFO [http_cmd]: http status code OK 22:53:04 DEBUG --- status code --- 22:53:04 INFO [http_cmd]: http status code OK 22:53:04 INFO [http_cmd]: http status code OK 22:53:04 DEBUG http status code is 200 (expected 200) 22:53:04 DEBUG --- status code --- 22:53:04 DEBUG --- status code --- 22:53:04 DEBUG --- http response --- 22:53:04 DEBUG http status code is 200 (expected 200) 22:53:04 DEBUG http status code is 200 (expected 200) 22:53:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:04 DEBUG --- http response --- 22:53:04 DEBUG --- http response --- 22:53:04 DEBUG ----- output ----- 22:53:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:53:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:04 DEBUG { 22:53:04 DEBUG ----- output ----- 22:53:04 DEBUG ----- output ----- 22:53:04 DEBUG "status": "success", 22:53:04 DEBUG { 22:53:04 DEBUG { 22:53:04 DEBUG "data": { 22:53:04 DEBUG "status": "success", 22:53:04 DEBUG "status": "success", 22:53:04 DEBUG "resultType": "vector", 22:53:04 DEBUG "data": { 22:53:04 DEBUG "data": { 22:53:04 DEBUG "result": [ 22:53:04 DEBUG "resultType": "vector", 22:53:04 DEBUG "resultType": "vector", 22:53:04 DEBUG { 22:53:04 DEBUG "result": [ 22:53:04 DEBUG "result": [ 22:53:04 DEBUG "metric": { 22:53:04 DEBUG { 22:53:04 DEBUG { 22:53:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:53:04 DEBUG "metric": { 22:53:04 DEBUG "metric": { 22:53:04 DEBUG }, 22:53:04 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:53:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:53:04 DEBUG "value": [ 22:53:04 DEBUG }, 22:53:04 DEBUG }, 22:53:04 DEBUG 22:53:04 DEBUG "value": [ 22:53:04 DEBUG "value": [ 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG [...] 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG [...] 22:53:04 DEBUG [...] 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:53:04 DEBUG 22:53:04 DEBUG 22:53:04 DEBUG }, 22:53:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:53:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:53:04 DEBUG "value": [ 22:53:04 DEBUG }, 22:53:04 DEBUG }, 22:53:04 DEBUG 1686693184, 22:53:04 DEBUG "value": [ 22:53:04 DEBUG "value": [ 22:53:04 DEBUG "0.2" 22:53:04 DEBUG 1686693184, 22:53:04 DEBUG 1686693184, 22:53:04 DEBUG ] 22:53:04 DEBUG "0" 22:53:04 DEBUG "0" 22:53:04 DEBUG } 22:53:04 DEBUG ] 22:53:04 DEBUG ] 22:53:04 DEBUG ] 22:53:04 DEBUG } 22:53:04 DEBUG } 22:53:04 DEBUG } 22:53:04 DEBUG ] 22:53:04 DEBUG ] 22:53:04 DEBUG } 22:53:04 DEBUG } 22:53:04 DEBUG } 22:53:04 DEBUG } 22:53:04 DEBUG } 22:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693184" 22:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693184" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "ds-idrepo-2" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "ds-idrepo-0" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693184, 22:53:05 DEBUG "633.8" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693184, 22:53:05 DEBUG "0" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693185" 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693185" 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "ds-cts-0" 22:53:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:53:05 DEBUG }, 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "ds-idrepo-0" 22:53:05 DEBUG 22:53:05 DEBUG }, 22:53:05 DEBUG "pod": "ds-idrepo-0" 22:53:05 DEBUG "value": [ 22:53:05 DEBUG }, 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG "633.8333333333334" 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG ] 22:53:05 DEBUG "6715.010556425797" 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG } 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "0" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "0" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "ds-cts-0" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "ds-idrepo-0" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "0" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "instance": "10.142.0.100:9100" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "instance": "10.142.0.99:9100" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "0" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "ds-cts-0" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "ds-cts-2" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "0" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "0" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "13615144960" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "ds-idrepo-2" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "ds-idrepo-0" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "633.8" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "instance": "10.142.0.100:9100" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "instance": "10.142.0.99:9100" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "0.0003333333333330302" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "instance": "10.142.0.100:9100" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "instance": "10.142.0.99:9100" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "21264461.7" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "instance": "10.142.0.100:9100" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "instance": "10.142.0.99:9100" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "21268815.43333333" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693185" 22:53:05 INFO [http_cmd]: http status code OK 22:53:05 DEBUG --- status code --- 22:53:05 DEBUG http status code is 200 (expected 200) 22:53:05 DEBUG --- http response --- 22:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:05 DEBUG ----- output ----- 22:53:05 DEBUG { 22:53:05 DEBUG "status": "success", 22:53:05 DEBUG "data": { 22:53:05 DEBUG "resultType": "vector", 22:53:05 DEBUG "result": [ 22:53:05 DEBUG { 22:53:05 DEBUG "metric": { 22:53:05 DEBUG "pod": "ds-idrepo-2" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG [...] 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG 22:53:05 DEBUG "pod": "ds-idrepo-0" 22:53:05 DEBUG }, 22:53:05 DEBUG "value": [ 22:53:05 DEBUG 1686693185, 22:53:05 DEBUG "633.8" 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG ] 22:53:05 DEBUG } 22:53:05 DEBUG } 22:53:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693186" 22:53:06 INFO [http_cmd]: http status code OK 22:53:06 DEBUG --- status code --- 22:53:06 DEBUG http status code is 200 (expected 200) 22:53:06 DEBUG --- http response --- 22:53:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:53:06 DEBUG ----- output ----- 22:53:06 DEBUG { 22:53:06 DEBUG "status": "success", 22:53:06 DEBUG "data": { 22:53:06 DEBUG "resultType": "vector", 22:53:06 DEBUG "result": [ 22:53:06 DEBUG { 22:53:06 DEBUG "metric": { 22:53:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:53:06 DEBUG }, 22:53:06 DEBUG "value": [ 22:53:06 DEBUG 22:53:06 DEBUG 22:53:06 DEBUG 22:53:06 DEBUG [...] 22:53:06 DEBUG 22:53:06 DEBUG 22:53:06 DEBUG 22:53:06 DEBUG "pod": "ds-idrepo-0" 22:53:06 DEBUG }, 22:53:06 DEBUG "value": [ 22:53:06 DEBUG 1686693186, 22:53:06 DEBUG "0.7491657018320506" 22:53:06 DEBUG ] 22:53:06 DEBUG } 22:53:06 DEBUG ] 22:53:06 DEBUG } 22:53:06 DEBUG } 22:53:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693186" 22:53:06 INFO [http_cmd]: http status code OK 22:53:06 DEBUG --- status code --- 22:53:06 DEBUG http status code is 200 (expected 200) 22:53:06 DEBUG --- http response --- 22:53:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:06 DEBUG ----- output ----- 22:53:06 DEBUG { 22:53:06 DEBUG "status": "success", 22:53:06 DEBUG "data": { 22:53:06 DEBUG "resultType": "vector", 22:53:06 DEBUG "result": [ 22:53:06 DEBUG { 22:53:06 DEBUG "metric": { 22:53:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:53:06 DEBUG }, 22:53:06 DEBUG "value": [ 22:53:06 DEBUG 22:53:06 DEBUG 22:53:06 DEBUG 22:53:06 DEBUG [...] 22:53:06 DEBUG 22:53:06 DEBUG 22:53:06 DEBUG 22:53:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:53:06 DEBUG }, 22:53:06 DEBUG "value": [ 22:53:06 DEBUG 1686693186, 22:53:06 DEBUG "0.2285149462107317" 22:53:06 DEBUG ] 22:53:06 DEBUG } 22:53:06 DEBUG ] 22:53:06 DEBUG } 22:53:06 DEBUG } 22:53:23 INFO 22:53:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:53:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:53:23 INFO [loop_until]: OK (rc = 0) 22:53:23 DEBUG --- stdout --- 22:53:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 11m 5760Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 10m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 10m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 2742m 13807Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 14m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3458m 4985Mi idm-58dc667486-rgz9j 2676m 5101Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 211m 644Mi 22:53:23 DEBUG --- stderr --- 22:53:23 DEBUG 22:53:27 INFO 22:53:27 INFO [loop_until]: kubectl --namespace=xlou top node 22:53:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:53:27 INFO [loop_until]: OK (rc = 0) 22:53:27 DEBUG --- stdout --- 22:53:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3355m 21% 5922Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 72m 0% 6861Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6829Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2600m 16% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 444m 2% 3131Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1203Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13726Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3076m 19% 14529Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14609Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 279m 1% 1963Mi 3% 22:53:27 DEBUG --- stderr --- 22:53:27 DEBUG 22:54:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693244" 22:54:04 INFO [http_cmd]: http status code OK 22:54:04 DEBUG --- status code --- 22:54:04 DEBUG http status code is 200 (expected 200) 22:54:04 DEBUG --- http response --- 22:54:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:04 DEBUG ----- output ----- 22:54:04 DEBUG { 22:54:04 DEBUG "status": "success", 22:54:04 DEBUG "data": { 22:54:04 DEBUG "resultType": "vector", 22:54:04 DEBUG "result": [ 22:54:04 DEBUG { 22:54:04 DEBUG "metric": { 22:54:04 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:54:04 DEBUG }, 22:54:04 DEBUG "value": [ 22:54:04 DEBUG 22:54:04 DEBUG 22:54:04 DEBUG 22:54:04 DEBUG [...] 22:54:04 DEBUG 22:54:04 DEBUG 22:54:04 DEBUG 22:54:04 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:54:04 DEBUG }, 22:54:04 DEBUG "value": [ 22:54:04 DEBUG 1686693244, 22:54:04 DEBUG "0.2" 22:54:04 DEBUG ] 22:54:04 DEBUG } 22:54:04 DEBUG ] 22:54:04 DEBUG } 22:54:04 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "ds-idrepo-2" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "ds-idrepo-0" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "653.0666666666666" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "0" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "0.000026066386989361913" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "0" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:05 DEBUG { 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG { 22:54:05 DEBUG { 22:54:05 DEBUG "data": { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "data": { 22:54:05 DEBUG "result": [ 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG { 22:54:05 DEBUG "result": [ 22:54:05 DEBUG "result": [ 22:54:05 DEBUG "metric": { 22:54:05 DEBUG { 22:54:05 DEBUG { 22:54:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG }, 22:54:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:54:05 DEBUG "pod": "ds-cts-0" 22:54:05 DEBUG "value": [ 22:54:05 DEBUG }, 22:54:05 DEBUG }, 22:54:05 DEBUG 22:54:05 DEBUG "value": [ 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG }, 22:54:05 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 22:54:05 DEBUG "pod": "ds-idrepo-0" 22:54:05 DEBUG "value": [ 22:54:05 DEBUG }, 22:54:05 DEBUG }, 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG "value": [ 22:54:05 DEBUG "0" 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG ] 22:54:05 DEBUG "0" 22:54:05 DEBUG "653.0666666666666" 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG ] 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "0" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "ds-cts-0" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "ds-idrepo-0" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "0" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "instance": "10.142.0.100:9100" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "instance": "10.142.0.99:9100" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "0" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "ds-cts-0" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "ds-cts-2" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "0" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "0" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "13616431104" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "ds-idrepo-2" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "ds-idrepo-0" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "653.0666666666666" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "instance": "10.142.0.100:9100" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "instance": "10.142.0.99:9100" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "0.00040000000000001514" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "instance": "10.142.0.100:9100" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "instance": "10.142.0.99:9100" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "19412048.3" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "instance": "10.142.0.100:9100" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "instance": "10.142.0.99:9100" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "19416235.8" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693245" 22:54:05 INFO [http_cmd]: http status code OK 22:54:05 DEBUG --- status code --- 22:54:05 DEBUG http status code is 200 (expected 200) 22:54:05 DEBUG --- http response --- 22:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:05 DEBUG ----- output ----- 22:54:05 DEBUG { 22:54:05 DEBUG "status": "success", 22:54:05 DEBUG "data": { 22:54:05 DEBUG "resultType": "vector", 22:54:05 DEBUG "result": [ 22:54:05 DEBUG { 22:54:05 DEBUG "metric": { 22:54:05 DEBUG "pod": "ds-idrepo-2" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG [...] 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG 22:54:05 DEBUG "pod": "ds-idrepo-0" 22:54:05 DEBUG }, 22:54:05 DEBUG "value": [ 22:54:05 DEBUG 1686693245, 22:54:05 DEBUG "653.1" 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG ] 22:54:05 DEBUG } 22:54:05 DEBUG } 22:54:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693246" 22:54:06 INFO [http_cmd]: http status code OK 22:54:06 DEBUG --- status code --- 22:54:06 DEBUG http status code is 200 (expected 200) 22:54:06 DEBUG --- http response --- 22:54:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:54:06 DEBUG ----- output ----- 22:54:06 DEBUG { 22:54:06 DEBUG "status": "success", 22:54:06 DEBUG "data": { 22:54:06 DEBUG "resultType": "vector", 22:54:06 DEBUG "result": [ 22:54:06 DEBUG { 22:54:06 DEBUG "metric": { 22:54:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:54:06 DEBUG }, 22:54:06 DEBUG "value": [ 22:54:06 DEBUG 22:54:06 DEBUG 22:54:06 DEBUG 22:54:06 DEBUG [...] 22:54:06 DEBUG 22:54:06 DEBUG 22:54:06 DEBUG 22:54:06 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 22:54:06 DEBUG }, 22:54:06 DEBUG "value": [ 22:54:06 DEBUG 1686693246, 22:54:06 DEBUG "0" 22:54:06 DEBUG ] 22:54:06 DEBUG } 22:54:06 DEBUG ] 22:54:06 DEBUG } 22:54:06 DEBUG } 22:54:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693246" 22:54:06 INFO [http_cmd]: http status code OK 22:54:06 DEBUG --- status code --- 22:54:06 DEBUG http status code is 200 (expected 200) 22:54:06 DEBUG --- http response --- 22:54:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:06 DEBUG ----- output ----- 22:54:06 DEBUG { 22:54:06 DEBUG "status": "success", 22:54:06 DEBUG "data": { 22:54:06 DEBUG "resultType": "vector", 22:54:06 DEBUG "result": [ 22:54:06 DEBUG { 22:54:06 DEBUG "metric": { 22:54:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:54:06 DEBUG }, 22:54:06 DEBUG "value": [ 22:54:06 DEBUG 22:54:06 DEBUG 22:54:06 DEBUG 22:54:06 DEBUG [...] 22:54:06 DEBUG 22:54:06 DEBUG 22:54:06 DEBUG 22:54:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:54:06 DEBUG }, 22:54:06 DEBUG "value": [ 22:54:06 DEBUG 1686693246, 22:54:06 DEBUG "0.23742458065000988" 22:54:06 DEBUG ] 22:54:06 DEBUG } 22:54:06 DEBUG ] 22:54:06 DEBUG } 22:54:06 DEBUG } 22:54:23 INFO 22:54:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:54:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:54:23 INFO [loop_until]: OK (rc = 0) 22:54:23 DEBUG --- stdout --- 22:54:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 10m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3210m 13818Mi ds-idrepo-1 17m 13821Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2973m 4986Mi idm-58dc667486-rgz9j 2501m 5102Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 196m 643Mi 22:54:23 DEBUG --- stderr --- 22:54:23 DEBUG 22:54:27 INFO 22:54:27 INFO [loop_until]: kubectl --namespace=xlou top node 22:54:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:54:27 INFO [loop_until]: OK (rc = 0) 22:54:27 DEBUG --- stdout --- 22:54:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2820m 17% 5933Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 73m 0% 6864Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6832Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2670m 16% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 437m 2% 3132Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 80m 0% 1484Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13729Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3171m 19% 14509Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 60m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14609Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 263m 1% 1960Mi 3% 22:54:27 DEBUG --- stderr --- 22:54:27 DEBUG 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0.2" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "ds-idrepo-2" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "ds-idrepo-0" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "610.3666666666667" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "idm-58dc667486-rgz9j" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:55:05 DEBUG { 22:55:05 DEBUG ----- output ----- 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG { 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG "data": { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "data": { 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG "result": [ 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:05 DEBUG { 22:55:05 DEBUG "result": [ 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG "metric": { 22:55:05 DEBUG { 22:55:05 DEBUG { 22:55:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG }, 22:55:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:55:05 DEBUG "data": { 22:55:05 DEBUG "value": [ 22:55:05 DEBUG }, 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG 22:55:05 DEBUG "value": [ 22:55:05 DEBUG "result": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG { 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "metric": { 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG "pod": "ds-cts-0" 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG }, 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG }, 22:55:05 DEBUG "pod": "lodemon-6455d5ff7c-zzndq" 22:55:05 DEBUG 22:55:05 DEBUG "value": [ 22:55:05 DEBUG }, 22:55:05 DEBUG [...] 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG "0" 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG 22:55:05 DEBUG ] 22:55:05 DEBUG "0" 22:55:05 DEBUG 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG "pod": "ds-idrepo-0" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG }, 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG "value": [ 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG } 22:55:05 DEBUG "610.3666666666667" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "idm-58dc667486-rgz9j" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "ds-cts-0" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "ds-idrepo-0" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "instance": "10.142.0.100:9100" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "instance": "10.142.0.99:9100" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "ds-cts-0" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "ds-cts-2" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "13616529408" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693305" 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "instance": "10.142.0.100:9100" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "instance": "10.142.0.99:9100" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "0.00026666666666604516" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "ds-idrepo-2" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "ds-idrepo-0" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "610.3" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "instance": "10.142.0.100:9100" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "instance": "10.142.0.99:9100" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "19082490" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "instance": "10.142.0.100:9100" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "instance": "10.142.0.99:9100" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "19086699.966666665" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693305" 22:55:05 INFO [http_cmd]: http status code OK 22:55:05 DEBUG --- status code --- 22:55:05 DEBUG http status code is 200 (expected 200) 22:55:05 DEBUG --- http response --- 22:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:05 DEBUG ----- output ----- 22:55:05 DEBUG { 22:55:05 DEBUG "status": "success", 22:55:05 DEBUG "data": { 22:55:05 DEBUG "resultType": "vector", 22:55:05 DEBUG "result": [ 22:55:05 DEBUG { 22:55:05 DEBUG "metric": { 22:55:05 DEBUG "pod": "ds-idrepo-2" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG [...] 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG 22:55:05 DEBUG "pod": "ds-idrepo-0" 22:55:05 DEBUG }, 22:55:05 DEBUG "value": [ 22:55:05 DEBUG 1686693305, 22:55:05 DEBUG "610.3" 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG ] 22:55:05 DEBUG } 22:55:05 DEBUG } 22:55:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693306" 22:55:06 INFO [http_cmd]: http status code OK 22:55:06 DEBUG --- status code --- 22:55:06 DEBUG http status code is 200 (expected 200) 22:55:06 DEBUG --- http response --- 22:55:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:55:06 DEBUG ----- output ----- 22:55:06 DEBUG { 22:55:06 DEBUG "status": "success", 22:55:06 DEBUG "data": { 22:55:06 DEBUG "resultType": "vector", 22:55:06 DEBUG "result": [ 22:55:06 DEBUG { 22:55:06 DEBUG "metric": { 22:55:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:55:06 DEBUG }, 22:55:06 DEBUG "value": [ 22:55:06 DEBUG 22:55:06 DEBUG 22:55:06 DEBUG 22:55:06 DEBUG [...] 22:55:06 DEBUG 22:55:06 DEBUG 22:55:06 DEBUG 22:55:06 DEBUG "pod": "lodemon-6455d5ff7c-zzndq" 22:55:06 DEBUG }, 22:55:06 DEBUG "value": [ 22:55:06 DEBUG 1686693306, 22:55:06 DEBUG "0" 22:55:06 DEBUG ] 22:55:06 DEBUG } 22:55:06 DEBUG ] 22:55:06 DEBUG } 22:55:06 DEBUG } 22:55:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693306" 22:55:06 INFO [http_cmd]: http status code OK 22:55:06 DEBUG --- status code --- 22:55:06 DEBUG http status code is 200 (expected 200) 22:55:06 DEBUG --- http response --- 22:55:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:06 DEBUG ----- output ----- 22:55:06 DEBUG { 22:55:06 DEBUG "status": "success", 22:55:06 DEBUG "data": { 22:55:06 DEBUG "resultType": "vector", 22:55:06 DEBUG "result": [ 22:55:06 DEBUG { 22:55:06 DEBUG "metric": { 22:55:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:55:06 DEBUG }, 22:55:06 DEBUG "value": [ 22:55:06 DEBUG 22:55:06 DEBUG 22:55:06 DEBUG 22:55:06 DEBUG [...] 22:55:06 DEBUG 22:55:06 DEBUG 22:55:06 DEBUG 22:55:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:55:06 DEBUG }, 22:55:06 DEBUG "value": [ 22:55:06 DEBUG 1686693306, 22:55:06 DEBUG "0.2298158834523235" 22:55:06 DEBUG ] 22:55:06 DEBUG } 22:55:06 DEBUG ] 22:55:06 DEBUG } 22:55:06 DEBUG } 22:55:23 INFO 22:55:23 INFO [loop_until]: kubectl --namespace=xlou top pods 22:55:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:55:24 INFO [loop_until]: OK (rc = 0) 22:55:24 DEBUG --- stdout --- 22:55:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5761Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 10m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3073m 13804Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2852m 4988Mi idm-58dc667486-rgz9j 2480m 5101Mi lodemon-6455d5ff7c-zzndq 3m 71Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 192m 643Mi 22:55:24 DEBUG --- stderr --- 22:55:24 DEBUG 22:55:27 INFO 22:55:27 INFO [loop_until]: kubectl --namespace=xlou top node 22:55:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:55:27 INFO [loop_until]: OK (rc = 0) 22:55:27 DEBUG --- stdout --- 22:55:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3048m 19% 5924Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6832Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2489m 15% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 73m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 433m 2% 3144Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1201Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 73m 0% 13727Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3122m 19% 14509Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14611Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 260m 1% 1963Mi 3% 22:55:27 DEBUG --- stderr --- 22:55:27 DEBUG 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:05 DEBUG { 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG { 22:56:05 DEBUG { 22:56:05 DEBUG "data": { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG "data": { 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "result": [ 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:56:05 DEBUG { 22:56:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:56:05 DEBUG }, 22:56:05 DEBUG "metric": { 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG }, 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:56:05 DEBUG 22:56:05 DEBUG "pod": "ds-idrepo-0" 22:56:05 DEBUG }, 22:56:05 DEBUG 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG }, 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "0" 22:56:05 DEBUG "value": [ 22:56:05 DEBUG "2.506159597449259" 22:56:05 DEBUG ] 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG "0.2" 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG ] 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "0" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "ds-idrepo-2" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "ds-idrepo-0" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "610" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "ds-cts-0" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "ds-idrepo-0" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "609.9666666666667" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "0" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "ds-idrepo-0" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "5419.910689030758" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "ds-idrepo-0" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "6737.467126637324" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "ds-cts-0" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "ds-idrepo-0" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "0" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "instance": "10.142.0.100:9100" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "instance": "10.142.0.99:9100" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "0" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "ds-cts-0" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "ds-cts-2" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "0" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "0" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "13617127424" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "pod": "ds-idrepo-2" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "pod": "ds-idrepo-0" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "610" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "instance": "10.142.0.100:9100" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "instance": "10.142.0.99:9100" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "0.00036666666666747006" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "instance": "10.142.0.100:9100" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "instance": "10.142.0.99:9100" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "19502082.633333333" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693365" 22:56:05 INFO [http_cmd]: http status code OK 22:56:05 DEBUG --- status code --- 22:56:05 DEBUG http status code is 200 (expected 200) 22:56:05 DEBUG --- http response --- 22:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:05 DEBUG ----- output ----- 22:56:05 DEBUG { 22:56:05 DEBUG "status": "success", 22:56:05 DEBUG "data": { 22:56:05 DEBUG "resultType": "vector", 22:56:05 DEBUG "result": [ 22:56:05 DEBUG { 22:56:05 DEBUG "metric": { 22:56:05 DEBUG "instance": "10.142.0.100:9100" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG [...] 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG 22:56:05 DEBUG "instance": "10.142.0.99:9100" 22:56:05 DEBUG }, 22:56:05 DEBUG "value": [ 22:56:05 DEBUG 1686693365, 22:56:05 DEBUG "19505748.033333335" 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG ] 22:56:05 DEBUG } 22:56:05 DEBUG } 22:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693365" 22:56:06 INFO [http_cmd]: http status code OK 22:56:06 DEBUG --- status code --- 22:56:06 DEBUG http status code is 200 (expected 200) 22:56:06 DEBUG --- http response --- 22:56:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:06 DEBUG ----- output ----- 22:56:06 DEBUG { 22:56:06 DEBUG "status": "success", 22:56:06 DEBUG "data": { 22:56:06 DEBUG "resultType": "vector", 22:56:06 DEBUG "result": [ 22:56:06 DEBUG { 22:56:06 DEBUG "metric": { 22:56:06 DEBUG "pod": "ds-idrepo-2" 22:56:06 DEBUG }, 22:56:06 DEBUG "value": [ 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG [...] 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG "pod": "ds-idrepo-0" 22:56:06 DEBUG }, 22:56:06 DEBUG "value": [ 22:56:06 DEBUG 1686693365, 22:56:06 DEBUG "610" 22:56:06 DEBUG ] 22:56:06 DEBUG } 22:56:06 DEBUG ] 22:56:06 DEBUG } 22:56:06 DEBUG } 22:56:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693366" 22:56:06 INFO [http_cmd]: http status code OK 22:56:06 DEBUG --- status code --- 22:56:06 DEBUG http status code is 200 (expected 200) 22:56:06 DEBUG --- http response --- 22:56:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:56:06 DEBUG ----- output ----- 22:56:06 DEBUG { 22:56:06 DEBUG "status": "success", 22:56:06 DEBUG "data": { 22:56:06 DEBUG "resultType": "vector", 22:56:06 DEBUG "result": [ 22:56:06 DEBUG { 22:56:06 DEBUG "metric": { 22:56:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:56:06 DEBUG }, 22:56:06 DEBUG "value": [ 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG [...] 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG "pod": "ds-idrepo-0" 22:56:06 DEBUG }, 22:56:06 DEBUG "value": [ 22:56:06 DEBUG 1686693366, 22:56:06 DEBUG "0.459394371968225" 22:56:06 DEBUG ] 22:56:06 DEBUG } 22:56:06 DEBUG ] 22:56:06 DEBUG } 22:56:06 DEBUG } 22:56:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693366" 22:56:06 INFO [http_cmd]: http status code OK 22:56:06 DEBUG --- status code --- 22:56:06 DEBUG http status code is 200 (expected 200) 22:56:06 DEBUG --- http response --- 22:56:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:06 DEBUG ----- output ----- 22:56:06 DEBUG { 22:56:06 DEBUG "status": "success", 22:56:06 DEBUG "data": { 22:56:06 DEBUG "resultType": "vector", 22:56:06 DEBUG "result": [ 22:56:06 DEBUG { 22:56:06 DEBUG "metric": { 22:56:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:56:06 DEBUG }, 22:56:06 DEBUG "value": [ 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG [...] 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG 22:56:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:56:06 DEBUG }, 22:56:06 DEBUG "value": [ 22:56:06 DEBUG 1686693366, 22:56:06 DEBUG "0.23025023150616414" 22:56:06 DEBUG ] 22:56:06 DEBUG } 22:56:06 DEBUG ] 22:56:06 DEBUG } 22:56:06 DEBUG } 22:56:24 INFO 22:56:24 INFO [loop_until]: kubectl --namespace=xlou top pods 22:56:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:56:24 INFO [loop_until]: OK (rc = 0) 22:56:24 DEBUG --- stdout --- 22:56:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 10m 5688Mi ds-cts-0 10m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3034m 13825Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2856m 4989Mi idm-58dc667486-rgz9j 2439m 5102Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 197m 644Mi 22:56:24 DEBUG --- stderr --- 22:56:24 DEBUG 22:56:28 INFO 22:56:28 INFO [loop_until]: kubectl --namespace=xlou top node 22:56:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:56:28 INFO [loop_until]: OK (rc = 0) 22:56:28 DEBUG --- stdout --- 22:56:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2981m 18% 5927Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6859Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6833Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2662m 16% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 469m 2% 3159Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 72m 0% 13729Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3226m 20% 14512Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14612Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 264m 1% 1965Mi 3% 22:56:28 DEBUG --- stderr --- 22:56:28 DEBUG 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:57:05 DEBUG "pod": "ds-cts-2" 22:57:05 DEBUG }, 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:57:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:57:05 DEBUG }, 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0" 22:57:05 DEBUG "0" 22:57:05 DEBUG ] 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0.2" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "ds-idrepo-2" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "ds-idrepo-0" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "571.3666666666667" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:57:05 DEBUG { 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG { 22:57:05 DEBUG "data": { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "result": [ 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG { 22:57:05 DEBUG "result": [ 22:57:05 DEBUG "metric": { 22:57:05 DEBUG { 22:57:05 DEBUG "pod": "ds-cts-0" 22:57:05 DEBUG "metric": { 22:57:05 DEBUG }, 22:57:05 DEBUG "pod": "ds-cts-2" 22:57:05 DEBUG "value": [ 22:57:05 DEBUG }, 22:57:05 DEBUG 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "ds-idrepo-0" 22:57:05 DEBUG 22:57:05 DEBUG }, 22:57:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:57:05 DEBUG "value": [ 22:57:05 DEBUG }, 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG "571.3333333333334" 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG ] 22:57:05 DEBUG "0" 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "ds-cts-2" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "ds-cts-0" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "ds-idrepo-0" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "instance": "10.142.0.100:9100" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "instance": "10.142.0.99:9100" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0.0003333333333333854" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "ds-cts-0" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "ds-cts-2" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "13619081216" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693425" 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "pod": "ds-idrepo-2" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "pod": "ds-idrepo-0" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "571.4" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "instance": "10.142.0.100:9100" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "instance": "10.142.0.99:9100" 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "0.00030000000000048505" 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693425" 22:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693425" 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 INFO [http_cmd]: http status code OK 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG --- status code --- 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG http status code is 200 (expected 200) 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG --- http response --- 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG ----- output ----- 22:57:05 DEBUG { 22:57:05 DEBUG { 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "status": "success", 22:57:05 DEBUG "data": { 22:57:05 DEBUG "data": { 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "resultType": "vector", 22:57:05 DEBUG "result": [ 22:57:05 DEBUG "result": [ 22:57:05 DEBUG { 22:57:05 DEBUG { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "metric": { 22:57:05 DEBUG "instance": "10.142.0.100:9100" 22:57:05 DEBUG "instance": "10.142.0.100:9100" 22:57:05 DEBUG }, 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG [...] 22:57:05 DEBUG [...] 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG 22:57:05 DEBUG "instance": "10.142.0.99:9100" 22:57:05 DEBUG "instance": "10.142.0.99:9100" 22:57:05 DEBUG }, 22:57:05 DEBUG }, 22:57:05 DEBUG "value": [ 22:57:05 DEBUG "value": [ 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG 1686693425, 22:57:05 DEBUG "18254388.966666665" 22:57:05 DEBUG "18256507.2" 22:57:05 DEBUG ] 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 DEBUG ] 22:57:05 DEBUG ] 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:05 DEBUG } 22:57:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693426" 22:57:06 INFO [http_cmd]: http status code OK 22:57:06 DEBUG --- status code --- 22:57:06 DEBUG http status code is 200 (expected 200) 22:57:06 DEBUG --- http response --- 22:57:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:06 DEBUG ----- output ----- 22:57:06 DEBUG { 22:57:06 DEBUG "status": "success", 22:57:06 DEBUG "data": { 22:57:06 DEBUG "resultType": "vector", 22:57:06 DEBUG "result": [ 22:57:06 DEBUG { 22:57:06 DEBUG "metric": { 22:57:06 DEBUG "pod": "ds-idrepo-2" 22:57:06 DEBUG }, 22:57:06 DEBUG "value": [ 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG [...] 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG "pod": "ds-idrepo-0" 22:57:06 DEBUG }, 22:57:06 DEBUG "value": [ 22:57:06 DEBUG 1686693426, 22:57:06 DEBUG "571.4" 22:57:06 DEBUG ] 22:57:06 DEBUG } 22:57:06 DEBUG ] 22:57:06 DEBUG } 22:57:06 DEBUG } 22:57:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693426" 22:57:06 INFO [http_cmd]: http status code OK 22:57:06 DEBUG --- status code --- 22:57:06 DEBUG http status code is 200 (expected 200) 22:57:06 DEBUG --- http response --- 22:57:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:57:06 DEBUG ----- output ----- 22:57:06 DEBUG { 22:57:06 DEBUG "status": "success", 22:57:06 DEBUG "data": { 22:57:06 DEBUG "resultType": "vector", 22:57:06 DEBUG "result": [ 22:57:06 DEBUG { 22:57:06 DEBUG "metric": { 22:57:06 DEBUG "pod": "ds-cts-2" 22:57:06 DEBUG }, 22:57:06 DEBUG "value": [ 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG [...] 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:57:06 DEBUG }, 22:57:06 DEBUG "value": [ 22:57:06 DEBUG 1686693426, 22:57:06 DEBUG "0" 22:57:06 DEBUG ] 22:57:06 DEBUG } 22:57:06 DEBUG ] 22:57:06 DEBUG } 22:57:06 DEBUG } 22:57:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693426" 22:57:06 INFO [http_cmd]: http status code OK 22:57:06 DEBUG --- status code --- 22:57:06 DEBUG http status code is 200 (expected 200) 22:57:06 DEBUG --- http response --- 22:57:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:06 DEBUG ----- output ----- 22:57:06 DEBUG { 22:57:06 DEBUG "status": "success", 22:57:06 DEBUG "data": { 22:57:06 DEBUG "resultType": "vector", 22:57:06 DEBUG "result": [ 22:57:06 DEBUG { 22:57:06 DEBUG "metric": { 22:57:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:57:06 DEBUG }, 22:57:06 DEBUG "value": [ 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG [...] 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG 22:57:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:57:06 DEBUG }, 22:57:06 DEBUG "value": [ 22:57:06 DEBUG 1686693426, 22:57:06 DEBUG "0.2219948550936705" 22:57:06 DEBUG ] 22:57:06 DEBUG } 22:57:06 DEBUG ] 22:57:06 DEBUG } 22:57:06 DEBUG } 22:57:24 INFO 22:57:24 INFO [loop_until]: kubectl --namespace=xlou top pods 22:57:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:57:24 INFO [loop_until]: OK (rc = 0) 22:57:24 DEBUG --- stdout --- 22:57:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 8m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 418Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3130m 13827Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2660m 4990Mi idm-58dc667486-rgz9j 2767m 5102Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 196m 644Mi 22:57:24 DEBUG --- stderr --- 22:57:24 DEBUG 22:57:28 INFO 22:57:28 INFO [loop_until]: kubectl --namespace=xlou top node 22:57:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:57:28 INFO [loop_until]: OK (rc = 0) 22:57:28 DEBUG --- stdout --- 22:57:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2926m 18% 5927Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6858Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6831Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2655m 16% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 72m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 451m 2% 3151Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1478Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 59m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1201Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 69m 0% 13731Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3159m 19% 14513Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14615Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 258m 1% 1967Mi 3% 22:57:28 DEBUG --- stderr --- 22:57:28 DEBUG 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:58:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:58:05 DEBUG }, 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:58:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:58:05 DEBUG }, 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0" 22:58:05 DEBUG "0.008924087950018644" 22:58:05 DEBUG ] 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0.2" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "ds-idrepo-2" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "ds-idrepo-0" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "621.2" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "ds-cts-0" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "ds-idrepo-0" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "621.2" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "ds-cts-0" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "ds-idrepo-0" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "instance": "10.142.0.100:9100" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "instance": "10.142.0.99:9100" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0.0003333333333333262" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "ds-cts-0" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "ds-cts-2" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "13619511296" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693485" 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "pod": "ds-idrepo-2" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "pod": "ds-idrepo-0" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "621.2" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "instance": "10.142.0.100:9100" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "instance": "10.142.0.99:9100" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "0.00026666666666604516" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "instance": "10.142.0.100:9100" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "instance": "10.142.0.99:9100" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "18465905.766666666" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693485" 22:58:05 INFO [http_cmd]: http status code OK 22:58:05 DEBUG --- status code --- 22:58:05 DEBUG http status code is 200 (expected 200) 22:58:05 DEBUG --- http response --- 22:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:05 DEBUG ----- output ----- 22:58:05 DEBUG { 22:58:05 DEBUG "status": "success", 22:58:05 DEBUG "data": { 22:58:05 DEBUG "resultType": "vector", 22:58:05 DEBUG "result": [ 22:58:05 DEBUG { 22:58:05 DEBUG "metric": { 22:58:05 DEBUG "instance": "10.142.0.100:9100" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG [...] 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG 22:58:05 DEBUG "instance": "10.142.0.99:9100" 22:58:05 DEBUG }, 22:58:05 DEBUG "value": [ 22:58:05 DEBUG 1686693485, 22:58:05 DEBUG "18461802.1" 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG ] 22:58:05 DEBUG } 22:58:05 DEBUG } 22:58:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693486" 22:58:06 INFO [http_cmd]: http status code OK 22:58:06 DEBUG --- status code --- 22:58:06 DEBUG http status code is 200 (expected 200) 22:58:06 DEBUG --- http response --- 22:58:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:06 DEBUG ----- output ----- 22:58:06 DEBUG { 22:58:06 DEBUG "status": "success", 22:58:06 DEBUG "data": { 22:58:06 DEBUG "resultType": "vector", 22:58:06 DEBUG "result": [ 22:58:06 DEBUG { 22:58:06 DEBUG "metric": { 22:58:06 DEBUG "pod": "ds-idrepo-2" 22:58:06 DEBUG }, 22:58:06 DEBUG "value": [ 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG [...] 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG "pod": "ds-idrepo-0" 22:58:06 DEBUG }, 22:58:06 DEBUG "value": [ 22:58:06 DEBUG 1686693486, 22:58:06 DEBUG "621.2333333333333" 22:58:06 DEBUG ] 22:58:06 DEBUG } 22:58:06 DEBUG ] 22:58:06 DEBUG } 22:58:06 DEBUG } 22:58:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693486" 22:58:06 INFO [http_cmd]: http status code OK 22:58:06 DEBUG --- status code --- 22:58:06 DEBUG http status code is 200 (expected 200) 22:58:06 DEBUG --- http response --- 22:58:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:58:06 DEBUG ----- output ----- 22:58:06 DEBUG { 22:58:06 DEBUG "status": "success", 22:58:06 DEBUG "data": { 22:58:06 DEBUG "resultType": "vector", 22:58:06 DEBUG "result": [ 22:58:06 DEBUG { 22:58:06 DEBUG "metric": { 22:58:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:58:06 DEBUG }, 22:58:06 DEBUG "value": [ 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG [...] 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 22:58:06 DEBUG }, 22:58:06 DEBUG "value": [ 22:58:06 DEBUG 1686693486, 22:58:06 DEBUG "0" 22:58:06 DEBUG ] 22:58:06 DEBUG } 22:58:06 DEBUG ] 22:58:06 DEBUG } 22:58:06 DEBUG } 22:58:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693486" 22:58:06 INFO [http_cmd]: http status code OK 22:58:06 DEBUG --- status code --- 22:58:06 DEBUG http status code is 200 (expected 200) 22:58:06 DEBUG --- http response --- 22:58:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:06 DEBUG ----- output ----- 22:58:06 DEBUG { 22:58:06 DEBUG "status": "success", 22:58:06 DEBUG "data": { 22:58:06 DEBUG "resultType": "vector", 22:58:06 DEBUG "result": [ 22:58:06 DEBUG { 22:58:06 DEBUG "metric": { 22:58:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:58:06 DEBUG }, 22:58:06 DEBUG "value": [ 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG [...] 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG 22:58:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:58:06 DEBUG }, 22:58:06 DEBUG "value": [ 22:58:06 DEBUG 1686693486, 22:58:06 DEBUG "0.22324353759527965" 22:58:06 DEBUG ] 22:58:06 DEBUG } 22:58:06 DEBUG ] 22:58:06 DEBUG } 22:58:06 DEBUG } 22:58:24 INFO 22:58:24 INFO [loop_until]: kubectl --namespace=xlou top pods 22:58:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:58:24 INFO [loop_until]: OK (rc = 0) 22:58:24 DEBUG --- stdout --- 22:58:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 10m 5712Mi am-7849cf7bdb-xrngh 9m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3141m 13825Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3048m 4994Mi idm-58dc667486-rgz9j 2654m 5101Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 214m 644Mi 22:58:24 DEBUG --- stderr --- 22:58:24 DEBUG 22:58:28 INFO 22:58:28 INFO [loop_until]: kubectl --namespace=xlou top node 22:58:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:58:28 INFO [loop_until]: OK (rc = 0) 22:58:28 DEBUG --- stdout --- 22:58:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3167m 19% 5930Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6829Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2688m 16% 6079Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 75m 0% 6952Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 452m 2% 3153Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 58m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1196Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 72m 0% 13730Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3212m 20% 14517Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 60m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14616Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 273m 1% 1968Mi 3% 22:58:28 DEBUG --- stderr --- 22:58:28 DEBUG 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:59:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:59:05 DEBUG }, 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:59:05 DEBUG "pod": "idm-58dc667486-rgz9j" 22:59:05 DEBUG }, 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "0" 22:59:05 DEBUG "3.0325986131362828" 22:59:05 DEBUG ] 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "ds-idrepo-2" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "ds-idrepo-0" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "655.4" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:59:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:59:05 DEBUG }, 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:59:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:59:05 DEBUG }, 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "0.2" 22:59:05 DEBUG "0" 22:59:05 DEBUG ] 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "idm-58dc667486-rgz9j" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "0" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "ds-cts-0" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "ds-idrepo-0" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "655.4" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "0" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "idm-58dc667486-rgz9j" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "436.96881334234206" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "ds-cts-0" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "ds-idrepo-0" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "0" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "instance": "10.142.0.100:9100" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "instance": "10.142.0.99:9100" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "0" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "ds-cts-0" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "ds-cts-2" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "0" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "0" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693545" 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "13615210496" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "instance": "10.142.0.100:9100" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "instance": "10.142.0.99:9100" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "0.00046666666666700016" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693545" 22:59:05 INFO [http_cmd]: http status code OK 22:59:05 DEBUG --- status code --- 22:59:05 DEBUG http status code is 200 (expected 200) 22:59:05 DEBUG --- http response --- 22:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:05 DEBUG ----- output ----- 22:59:05 DEBUG { 22:59:05 DEBUG "status": "success", 22:59:05 DEBUG "data": { 22:59:05 DEBUG "resultType": "vector", 22:59:05 DEBUG "result": [ 22:59:05 DEBUG { 22:59:05 DEBUG "metric": { 22:59:05 DEBUG "pod": "ds-idrepo-2" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG [...] 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG 22:59:05 DEBUG "pod": "ds-idrepo-0" 22:59:05 DEBUG }, 22:59:05 DEBUG "value": [ 22:59:05 DEBUG 1686693545, 22:59:05 DEBUG "655.4333333333333" 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG ] 22:59:05 DEBUG } 22:59:05 DEBUG } 22:59:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693546" 22:59:06 INFO [http_cmd]: http status code OK 22:59:06 DEBUG --- status code --- 22:59:06 DEBUG http status code is 200 (expected 200) 22:59:06 DEBUG --- http response --- 22:59:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:06 DEBUG ----- output ----- 22:59:06 DEBUG { 22:59:06 DEBUG "status": "success", 22:59:06 DEBUG "data": { 22:59:06 DEBUG "resultType": "vector", 22:59:06 DEBUG "result": [ 22:59:06 DEBUG { 22:59:06 DEBUG "metric": { 22:59:06 DEBUG "instance": "10.142.0.100:9100" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG [...] 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG "instance": "10.142.0.99:9100" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 1686693546, 22:59:06 DEBUG "19481838.96666667" 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG } 22:59:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693546" 22:59:06 INFO [http_cmd]: http status code OK 22:59:06 DEBUG --- status code --- 22:59:06 DEBUG http status code is 200 (expected 200) 22:59:06 DEBUG --- http response --- 22:59:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:06 DEBUG ----- output ----- 22:59:06 DEBUG { 22:59:06 DEBUG "status": "success", 22:59:06 DEBUG "data": { 22:59:06 DEBUG "resultType": "vector", 22:59:06 DEBUG "result": [ 22:59:06 DEBUG { 22:59:06 DEBUG "metric": { 22:59:06 DEBUG "instance": "10.142.0.100:9100" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG [...] 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG "instance": "10.142.0.99:9100" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 1686693546, 22:59:06 DEBUG "19477586.4" 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG } 22:59:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693546" 22:59:06 INFO [http_cmd]: http status code OK 22:59:06 DEBUG --- status code --- 22:59:06 DEBUG http status code is 200 (expected 200) 22:59:06 DEBUG --- http response --- 22:59:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:06 DEBUG ----- output ----- 22:59:06 DEBUG { 22:59:06 DEBUG "status": "success", 22:59:06 DEBUG "data": { 22:59:06 DEBUG "resultType": "vector", 22:59:06 DEBUG "result": [ 22:59:06 DEBUG { 22:59:06 DEBUG "metric": { 22:59:06 DEBUG "pod": "ds-idrepo-2" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG [...] 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG "pod": "ds-idrepo-0" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 1686693546, 22:59:06 DEBUG "655.4333333333333" 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG } 22:59:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693546" 22:59:06 INFO [http_cmd]: http status code OK 22:59:06 DEBUG --- status code --- 22:59:06 DEBUG http status code is 200 (expected 200) 22:59:06 DEBUG --- http response --- 22:59:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:59:06 DEBUG ----- output ----- 22:59:06 DEBUG { 22:59:06 DEBUG "status": "success", 22:59:06 DEBUG "data": { 22:59:06 DEBUG "resultType": "vector", 22:59:06 DEBUG "result": [ 22:59:06 DEBUG { 22:59:06 DEBUG "metric": { 22:59:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG [...] 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG "pod": "idm-58dc667486-rgz9j" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 1686693546, 22:59:06 DEBUG "0" 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG } 22:59:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693546" 22:59:06 INFO [http_cmd]: http status code OK 22:59:06 DEBUG --- status code --- 22:59:06 DEBUG http status code is 200 (expected 200) 22:59:06 DEBUG --- http response --- 22:59:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:06 DEBUG ----- output ----- 22:59:06 DEBUG { 22:59:06 DEBUG "status": "success", 22:59:06 DEBUG "data": { 22:59:06 DEBUG "resultType": "vector", 22:59:06 DEBUG "result": [ 22:59:06 DEBUG { 22:59:06 DEBUG "metric": { 22:59:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG [...] 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG 22:59:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 22:59:06 DEBUG }, 22:59:06 DEBUG "value": [ 22:59:06 DEBUG 1686693546, 22:59:06 DEBUG "0.2359653674030381" 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG ] 22:59:06 DEBUG } 22:59:06 DEBUG } 22:59:24 INFO 22:59:24 INFO [loop_until]: kubectl --namespace=xlou top pods 22:59:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:59:24 INFO [loop_until]: OK (rc = 0) 22:59:24 DEBUG --- stdout --- 22:59:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 11m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 13m 5688Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3284m 13802Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3057m 4996Mi idm-58dc667486-rgz9j 2917m 5102Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 219m 643Mi 22:59:24 DEBUG --- stderr --- 22:59:24 DEBUG 22:59:28 INFO 22:59:28 INFO [loop_until]: kubectl --namespace=xlou top node 22:59:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:59:28 INFO [loop_until]: OK (rc = 0) 22:59:28 DEBUG --- stdout --- 22:59:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3185m 20% 5932Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 73m 0% 6864Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 77m 0% 6831Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2798m 17% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 77m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 450m 2% 3149Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 83m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13731Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3437m 21% 14532Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14610Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 262m 1% 1966Mi 3% 22:59:28 DEBUG --- stderr --- 22:59:28 DEBUG 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG { 23:00:05 DEBUG { 23:00:05 DEBUG { 23:00:05 DEBUG { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "data": { 23:00:05 DEBUG "data": { 23:00:05 DEBUG "data": { 23:00:05 DEBUG "data": { 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "result": [ 23:00:05 DEBUG "result": [ 23:00:05 DEBUG "result": [ 23:00:05 DEBUG "result": [ 23:00:05 DEBUG { 23:00:05 DEBUG { 23:00:05 DEBUG { 23:00:05 DEBUG { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:00:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:00:05 DEBUG "pod": "ds-idrepo-2" 23:00:05 DEBUG "pod": "ds-cts-2" 23:00:05 DEBUG }, 23:00:05 DEBUG }, 23:00:05 DEBUG }, 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG "value": [ 23:00:05 DEBUG "value": [ 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG [...] 23:00:05 DEBUG [...] 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:00:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:00:05 DEBUG "pod": "ds-idrepo-0" 23:00:05 DEBUG "pod": "idm-58dc667486-rgz9j" 23:00:05 DEBUG }, 23:00:05 DEBUG }, 23:00:05 DEBUG }, 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG "value": [ 23:00:05 DEBUG "value": [ 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "0" 23:00:05 DEBUG "0" 23:00:05 DEBUG "574.0666666666666" 23:00:05 DEBUG "0" 23:00:05 DEBUG ] 23:00:05 DEBUG ] 23:00:05 DEBUG ] 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG ] 23:00:05 DEBUG ] 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "data": { 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "result": [ 23:00:05 DEBUG { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "pod": "ds-cts-2" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "0" 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "data": { 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "result": [ 23:00:05 DEBUG { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "0.2" 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "data": { 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "result": [ 23:00:05 DEBUG { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "pod": "ds-cts-0" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "pod": "ds-idrepo-0" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "574.0666666666666" 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG { 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG "data": { 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG "result": [ 23:00:05 DEBUG { 23:00:05 DEBUG { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "data": { 23:00:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG }, 23:00:05 DEBUG "result": [ 23:00:05 DEBUG "value": [ 23:00:05 DEBUG { 23:00:05 DEBUG 23:00:05 DEBUG "metric": { 23:00:05 DEBUG 23:00:05 DEBUG "pod": "ds-cts-2" 23:00:05 DEBUG 23:00:05 DEBUG }, 23:00:05 DEBUG [...] 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:00:05 DEBUG [...] 23:00:05 DEBUG }, 23:00:05 DEBUG 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG 23:00:05 DEBUG "0" 23:00:05 DEBUG "pod": "idm-58dc667486-rgz9j" 23:00:05 DEBUG ] 23:00:05 DEBUG }, 23:00:05 DEBUG } 23:00:05 DEBUG "value": [ 23:00:05 DEBUG ] 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG } 23:00:05 DEBUG "0" 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693605" 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "data": { 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "result": [ 23:00:05 DEBUG { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "instance": "10.142.0.100:9100" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "instance": "10.142.0.99:9100" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "0" 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "data": { 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "result": [ 23:00:05 DEBUG { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "pod": "ds-cts-0" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "pod": "ds-idrepo-0" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "0" 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "data": { 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "result": [ 23:00:05 DEBUG { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "pod": "ds-cts-0" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "pod": "ds-cts-2" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "0" 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "data": { 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "result": [ 23:00:05 DEBUG { 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "0" 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693605" 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693605" 23:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693605" 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG --- status code --- 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:05 DEBUG { 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG { 23:00:05 DEBUG "data": { 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG "data": { 23:00:05 DEBUG "result": [ 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG { 23:00:05 DEBUG "result": [ 23:00:05 DEBUG "metric": { 23:00:05 DEBUG { 23:00:05 DEBUG "instance": "10.142.0.100:9100" 23:00:05 DEBUG "metric": { 23:00:05 DEBUG }, 23:00:05 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:00:05 DEBUG "value": [ 23:00:05 DEBUG }, 23:00:05 DEBUG 23:00:05 INFO [http_cmd]: http status code OK 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 23:00:05 DEBUG --- status code --- 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG http status code is 200 (expected 200) 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG --- http response --- 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG ----- output ----- 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG { 23:00:05 DEBUG 23:00:05 DEBUG "instance": "10.142.0.99:9100" 23:00:05 DEBUG "status": "success", 23:00:05 DEBUG 23:00:05 DEBUG }, 23:00:05 DEBUG "data": { 23:00:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:00:05 DEBUG "value": [ 23:00:05 DEBUG "resultType": "vector", 23:00:05 DEBUG }, 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "result": [ 23:00:05 DEBUG "value": [ 23:00:05 DEBUG "0.00030000000000048505" 23:00:05 DEBUG { 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG ] 23:00:05 DEBUG "metric": { 23:00:05 DEBUG "13616803840" 23:00:05 DEBUG } 23:00:05 DEBUG "pod": "ds-idrepo-2" 23:00:05 DEBUG ] 23:00:05 DEBUG ] 23:00:05 DEBUG }, 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:05 DEBUG "value": [ 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG 23:00:05 DEBUG } 23:00:05 DEBUG 23:00:05 DEBUG } 23:00:05 DEBUG 23:00:05 DEBUG [...] 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG 23:00:05 DEBUG "pod": "ds-idrepo-0" 23:00:05 DEBUG }, 23:00:05 DEBUG "value": [ 23:00:05 DEBUG 1686693605, 23:00:05 DEBUG "574.0666666666666" 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG ] 23:00:05 DEBUG } 23:00:05 DEBUG } 23:00:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693606" 23:00:06 INFO [http_cmd]: http status code OK 23:00:06 DEBUG --- status code --- 23:00:06 DEBUG http status code is 200 (expected 200) 23:00:06 DEBUG --- http response --- 23:00:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:06 DEBUG ----- output ----- 23:00:06 DEBUG { 23:00:06 DEBUG "status": "success", 23:00:06 DEBUG "data": { 23:00:06 DEBUG "resultType": "vector", 23:00:06 DEBUG "result": [ 23:00:06 DEBUG { 23:00:06 DEBUG "metric": { 23:00:06 DEBUG "instance": "10.142.0.100:9100" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG [...] 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG "instance": "10.142.0.99:9100" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 1686693606, 23:00:06 DEBUG "13099783.399999999" 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG } 23:00:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693606" 23:00:06 INFO [http_cmd]: http status code OK 23:00:06 DEBUG --- status code --- 23:00:06 DEBUG http status code is 200 (expected 200) 23:00:06 DEBUG --- http response --- 23:00:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:06 DEBUG ----- output ----- 23:00:06 DEBUG { 23:00:06 DEBUG "status": "success", 23:00:06 DEBUG "data": { 23:00:06 DEBUG "resultType": "vector", 23:00:06 DEBUG "result": [ 23:00:06 DEBUG { 23:00:06 DEBUG "metric": { 23:00:06 DEBUG "instance": "10.142.0.100:9100" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG [...] 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG "instance": "10.142.0.99:9100" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 1686693606, 23:00:06 DEBUG "13095929.133333333" 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG } 23:00:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693606" 23:00:06 INFO [http_cmd]: http status code OK 23:00:06 DEBUG --- status code --- 23:00:06 DEBUG http status code is 200 (expected 200) 23:00:06 DEBUG --- http response --- 23:00:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:06 DEBUG ----- output ----- 23:00:06 DEBUG { 23:00:06 DEBUG "status": "success", 23:00:06 DEBUG "data": { 23:00:06 DEBUG "resultType": "vector", 23:00:06 DEBUG "result": [ 23:00:06 DEBUG { 23:00:06 DEBUG "metric": { 23:00:06 DEBUG "pod": "ds-idrepo-2" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG [...] 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG "pod": "ds-idrepo-0" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 1686693606, 23:00:06 DEBUG "574.0666666666666" 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG } 23:00:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693606" 23:00:06 INFO [http_cmd]: http status code OK 23:00:06 DEBUG --- status code --- 23:00:06 DEBUG http status code is 200 (expected 200) 23:00:06 DEBUG --- http response --- 23:00:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:00:06 DEBUG ----- output ----- 23:00:06 DEBUG { 23:00:06 DEBUG "status": "success", 23:00:06 DEBUG "data": { 23:00:06 DEBUG "resultType": "vector", 23:00:06 DEBUG "result": [ 23:00:06 DEBUG { 23:00:06 DEBUG "metric": { 23:00:06 DEBUG "pod": "ds-cts-2" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG [...] 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 1686693606, 23:00:06 DEBUG "0" 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG } 23:00:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693606" 23:00:06 INFO [http_cmd]: http status code OK 23:00:06 DEBUG --- status code --- 23:00:06 DEBUG http status code is 200 (expected 200) 23:00:06 DEBUG --- http response --- 23:00:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:06 DEBUG ----- output ----- 23:00:06 DEBUG { 23:00:06 DEBUG "status": "success", 23:00:06 DEBUG "data": { 23:00:06 DEBUG "resultType": "vector", 23:00:06 DEBUG "result": [ 23:00:06 DEBUG { 23:00:06 DEBUG "metric": { 23:00:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG [...] 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG 23:00:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:00:06 DEBUG }, 23:00:06 DEBUG "value": [ 23:00:06 DEBUG 1686693606, 23:00:06 DEBUG "0.22175631116535577" 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG ] 23:00:06 DEBUG } 23:00:06 DEBUG } 23:00:24 INFO 23:00:24 INFO [loop_until]: kubectl --namespace=xlou top pods 23:00:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:00:24 INFO [loop_until]: OK (rc = 0) 23:00:24 DEBUG --- stdout --- 23:00:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 12m 5712Mi am-7849cf7bdb-xrngh 9m 5690Mi ds-cts-0 10m 417Mi ds-cts-1 10m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 2536m 13822Mi ds-idrepo-1 18m 13820Mi ds-idrepo-2 13m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2359m 4995Mi idm-58dc667486-rgz9j 1845m 5102Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 146m 644Mi 23:00:24 DEBUG --- stderr --- 23:00:24 DEBUG 23:00:28 INFO 23:00:28 INFO [loop_until]: kubectl --namespace=xlou top node 23:00:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:00:28 INFO [loop_until]: OK (rc = 0) 23:00:28 DEBUG --- stdout --- 23:00:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2045m 12% 5935Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 75m 0% 6832Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 1728m 10% 6076Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 74m 0% 6952Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 361m 2% 3139Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 62m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1197Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13730Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 2145m 13% 14513Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 75m 0% 14610Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 213m 1% 1966Mi 3% 23:00:28 DEBUG --- stderr --- 23:00:28 DEBUG 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "0" 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "pod": "ds-cts-2" 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "pod": "ds-cts-2" 23:01:05 DEBUG }, 23:01:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 23:01:05 DEBUG 23:01:05 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 23:01:05 DEBUG }, 23:01:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "0" 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "0" 23:01:05 DEBUG ] 23:01:05 DEBUG "0.2" 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "pod": "ds-idrepo-2" 23:01:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:01:05 DEBUG }, 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG "pod": "ds-idrepo-0" 23:01:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:01:05 DEBUG }, 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "581.5" 23:01:05 DEBUG "0" 23:01:05 DEBUG ] 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "pod": "ds-cts-0" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG "pod": "ds-idrepo-0" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "581.5333333333333" 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "pod": "ds-cts-2" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG [...] 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:05 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG }, 23:01:05 DEBUG { 23:01:05 DEBUG "value": [ 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "data": { 23:01:05 DEBUG "0" 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG ] 23:01:05 DEBUG "result": [ 23:01:05 DEBUG } 23:01:05 DEBUG { 23:01:05 DEBUG ] 23:01:05 DEBUG "metric": { 23:01:05 DEBUG } 23:01:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:01:05 DEBUG } 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "0" 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693665" 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "instance": "10.142.0.100:9100" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG "instance": "10.142.0.99:9100" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "0" 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "pod": "ds-cts-0" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG "pod": "ds-idrepo-0" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "0" 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "pod": "ds-cts-0" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG "pod": "ds-cts-2" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "0" 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693665" 23:01:05 INFO [http_cmd]: http status code OK 23:01:05 DEBUG --- status code --- 23:01:05 DEBUG http status code is 200 (expected 200) 23:01:05 DEBUG --- http response --- 23:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:05 DEBUG ----- output ----- 23:01:05 DEBUG { 23:01:05 DEBUG "status": "success", 23:01:05 DEBUG "data": { 23:01:05 DEBUG "resultType": "vector", 23:01:05 DEBUG "result": [ 23:01:05 DEBUG { 23:01:05 DEBUG "metric": { 23:01:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG [...] 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG 23:01:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:01:05 DEBUG }, 23:01:05 DEBUG "value": [ 23:01:05 DEBUG 1686693665, 23:01:05 DEBUG "0" 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG ] 23:01:05 DEBUG } 23:01:05 DEBUG } 23:01:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693666" 23:01:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693666" 23:01:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693666" 23:01:06 INFO [http_cmd]: http status code OK 23:01:06 INFO [http_cmd]: http status code OK 23:01:06 DEBUG --- status code --- 23:01:06 DEBUG --- status code --- 23:01:06 DEBUG http status code is 200 (expected 200) 23:01:06 DEBUG http status code is 200 (expected 200) 23:01:06 DEBUG --- http response --- 23:01:06 DEBUG --- http response --- 23:01:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:06 DEBUG ----- output ----- 23:01:06 DEBUG ----- output ----- 23:01:06 DEBUG { 23:01:06 DEBUG { 23:01:06 DEBUG "status": "success", 23:01:06 DEBUG "status": "success", 23:01:06 DEBUG "data": { 23:01:06 DEBUG "data": { 23:01:06 DEBUG "resultType": "vector", 23:01:06 DEBUG "resultType": "vector", 23:01:06 DEBUG "result": [ 23:01:06 DEBUG "result": [ 23:01:06 DEBUG { 23:01:06 DEBUG { 23:01:06 DEBUG "metric": { 23:01:06 DEBUG "metric": { 23:01:06 DEBUG "pod": "ds-idrepo-2" 23:01:06 DEBUG "instance": "10.142.0.100:9100" 23:01:06 DEBUG }, 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG "value": [ 23:01:06 INFO [http_cmd]: http status code OK 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG --- status code --- 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG http status code is 200 (expected 200) 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG --- http response --- 23:01:06 DEBUG [...] 23:01:06 DEBUG [...] 23:01:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG ----- output ----- 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG { 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG "status": "success", 23:01:06 DEBUG "pod": "ds-idrepo-0" 23:01:06 DEBUG "instance": "10.142.0.99:9100" 23:01:06 DEBUG "data": { 23:01:06 DEBUG }, 23:01:06 DEBUG }, 23:01:06 DEBUG "resultType": "vector", 23:01:06 DEBUG "value": [ 23:01:06 DEBUG "value": [ 23:01:06 DEBUG "result": [ 23:01:06 DEBUG 1686693666, 23:01:06 DEBUG 1686693666, 23:01:06 DEBUG { 23:01:06 DEBUG "581.4666666666667" 23:01:06 DEBUG "0.00036666666666557526" 23:01:06 DEBUG "metric": { 23:01:06 DEBUG ] 23:01:06 DEBUG ] 23:01:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:01:06 DEBUG } 23:01:06 DEBUG } 23:01:06 DEBUG }, 23:01:06 DEBUG ] 23:01:06 DEBUG ] 23:01:06 DEBUG "value": [ 23:01:06 DEBUG } 23:01:06 DEBUG } 23:01:06 DEBUG 23:01:06 DEBUG } 23:01:06 DEBUG } 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG [...] 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 1686693666, 23:01:06 DEBUG "13617356800" 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG } 23:01:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693666" 23:01:06 INFO [http_cmd]: http status code OK 23:01:06 DEBUG --- status code --- 23:01:06 DEBUG http status code is 200 (expected 200) 23:01:06 DEBUG --- http response --- 23:01:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:06 DEBUG ----- output ----- 23:01:06 DEBUG { 23:01:06 DEBUG "status": "success", 23:01:06 DEBUG "data": { 23:01:06 DEBUG "resultType": "vector", 23:01:06 DEBUG "result": [ 23:01:06 DEBUG { 23:01:06 DEBUG "metric": { 23:01:06 DEBUG "instance": "10.142.0.100:9100" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG [...] 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG "instance": "10.142.0.99:9100" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 1686693666, 23:01:06 DEBUG "19920185.466666665" 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG } 23:01:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693666" 23:01:06 INFO [http_cmd]: http status code OK 23:01:06 DEBUG --- status code --- 23:01:06 DEBUG http status code is 200 (expected 200) 23:01:06 DEBUG --- http response --- 23:01:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:06 DEBUG ----- output ----- 23:01:06 DEBUG { 23:01:06 DEBUG "status": "success", 23:01:06 DEBUG "data": { 23:01:06 DEBUG "resultType": "vector", 23:01:06 DEBUG "result": [ 23:01:06 DEBUG { 23:01:06 DEBUG "metric": { 23:01:06 DEBUG "instance": "10.142.0.100:9100" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG [...] 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG "instance": "10.142.0.99:9100" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 1686693666, 23:01:06 DEBUG "19916077.8" 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG } 23:01:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693666" 23:01:06 INFO [http_cmd]: http status code OK 23:01:06 DEBUG --- status code --- 23:01:06 DEBUG http status code is 200 (expected 200) 23:01:06 DEBUG --- http response --- 23:01:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:06 DEBUG ----- output ----- 23:01:06 DEBUG { 23:01:06 DEBUG "status": "success", 23:01:06 DEBUG "data": { 23:01:06 DEBUG "resultType": "vector", 23:01:06 DEBUG "result": [ 23:01:06 DEBUG { 23:01:06 DEBUG "metric": { 23:01:06 DEBUG "pod": "ds-idrepo-2" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG [...] 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG "pod": "ds-idrepo-0" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 1686693666, 23:01:06 DEBUG "581.4666666666667" 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG } 23:01:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693666" 23:01:06 INFO [http_cmd]: http status code OK 23:01:06 DEBUG --- status code --- 23:01:06 DEBUG http status code is 200 (expected 200) 23:01:06 DEBUG --- http response --- 23:01:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:01:06 DEBUG ----- output ----- 23:01:06 DEBUG { 23:01:06 DEBUG "status": "success", 23:01:06 DEBUG "data": { 23:01:06 DEBUG "resultType": "vector", 23:01:06 DEBUG "result": [ 23:01:06 DEBUG { 23:01:06 DEBUG "metric": { 23:01:06 DEBUG "pod": "ds-cts-2" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG [...] 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 1686693666, 23:01:06 DEBUG "0" 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG } 23:01:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693666" 23:01:06 INFO [http_cmd]: http status code OK 23:01:06 DEBUG --- status code --- 23:01:06 DEBUG http status code is 200 (expected 200) 23:01:06 DEBUG --- http response --- 23:01:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:06 DEBUG ----- output ----- 23:01:06 DEBUG { 23:01:06 DEBUG "status": "success", 23:01:06 DEBUG "data": { 23:01:06 DEBUG "resultType": "vector", 23:01:06 DEBUG "result": [ 23:01:06 DEBUG { 23:01:06 DEBUG "metric": { 23:01:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG [...] 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG 23:01:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:01:06 DEBUG }, 23:01:06 DEBUG "value": [ 23:01:06 DEBUG 1686693666, 23:01:06 DEBUG "0.18870591865779204" 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG ] 23:01:06 DEBUG } 23:01:06 DEBUG } 23:01:24 INFO 23:01:24 INFO [loop_until]: kubectl --namespace=xlou top pods 23:01:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:01:24 INFO [loop_until]: OK (rc = 0) 23:01:24 DEBUG --- stdout --- 23:01:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 6m 5690Mi ds-cts-0 11m 417Mi ds-cts-1 8m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3149m 13801Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3246m 5001Mi idm-58dc667486-rgz9j 2492m 5102Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 199m 645Mi 23:01:24 DEBUG --- stderr --- 23:01:24 DEBUG 23:01:28 INFO 23:01:28 INFO [loop_until]: kubectl --namespace=xlou top node 23:01:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:01:28 INFO [loop_until]: OK (rc = 0) 23:01:28 DEBUG --- stdout --- 23:01:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3308m 20% 5937Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6863Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6831Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2506m 15% 6080Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6950Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 429m 2% 3147Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1164Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13730Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3255m 20% 14514Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 64m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 71m 0% 14617Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 269m 1% 1964Mi 3% 23:01:28 DEBUG --- stderr --- 23:01:28 DEBUG 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "ds-idrepo-2" 23:02:05 DEBUG "metric": { 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:02:05 DEBUG }, 23:02:05 DEBUG "pod": "ds-cts-2" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "ds-idrepo-0" 23:02:05 DEBUG 23:02:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:02:05 DEBUG }, 23:02:05 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "660.4" 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0.2" 23:02:05 DEBUG ] 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "ds-cts-2" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "ds-cts-0" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "ds-idrepo-0" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "660.4" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693725" 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "instance": "10.142.0.100:9100" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "instance": "10.142.0.99:9100" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "ds-cts-2" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "ds-cts-0" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "ds-idrepo-0" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "ds-cts-0" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "ds-cts-2" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693725" 23:02:05 INFO [http_cmd]: http status code OK 23:02:05 DEBUG --- status code --- 23:02:05 DEBUG http status code is 200 (expected 200) 23:02:05 DEBUG --- http response --- 23:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:05 DEBUG ----- output ----- 23:02:05 DEBUG { 23:02:05 DEBUG "status": "success", 23:02:05 DEBUG "data": { 23:02:05 DEBUG "resultType": "vector", 23:02:05 DEBUG "result": [ 23:02:05 DEBUG { 23:02:05 DEBUG "metric": { 23:02:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG [...] 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG 23:02:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:02:05 DEBUG }, 23:02:05 DEBUG "value": [ 23:02:05 DEBUG 1686693725, 23:02:05 DEBUG "0" 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG ] 23:02:05 DEBUG } 23:02:05 DEBUG } 23:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693726" 23:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693726" 23:02:06 INFO [http_cmd]: http status code OK 23:02:06 DEBUG --- status code --- 23:02:06 DEBUG http status code is 200 (expected 200) 23:02:06 DEBUG --- http response --- 23:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:06 DEBUG ----- output ----- 23:02:06 DEBUG { 23:02:06 DEBUG "status": "success", 23:02:06 DEBUG "data": { 23:02:06 DEBUG "resultType": "vector", 23:02:06 DEBUG "result": [ 23:02:06 DEBUG { 23:02:06 DEBUG "metric": { 23:02:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG [...] 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 1686693726, 23:02:06 DEBUG "13618434048" 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG } 23:02:06 INFO [http_cmd]: http status code OK 23:02:06 DEBUG --- status code --- 23:02:06 DEBUG http status code is 200 (expected 200) 23:02:06 DEBUG --- http response --- 23:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:06 DEBUG ----- output ----- 23:02:06 DEBUG { 23:02:06 DEBUG "status": "success", 23:02:06 DEBUG "data": { 23:02:06 DEBUG "resultType": "vector", 23:02:06 DEBUG "result": [ 23:02:06 DEBUG { 23:02:06 DEBUG "metric": { 23:02:06 DEBUG "pod": "ds-idrepo-2" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG [...] 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG "pod": "ds-idrepo-0" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 1686693726, 23:02:06 DEBUG "660.3666666666667" 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG } 23:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693726" 23:02:06 INFO [http_cmd]: http status code OK 23:02:06 DEBUG --- status code --- 23:02:06 DEBUG http status code is 200 (expected 200) 23:02:06 DEBUG --- http response --- 23:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:06 DEBUG ----- output ----- 23:02:06 DEBUG { 23:02:06 DEBUG "status": "success", 23:02:06 DEBUG "data": { 23:02:06 DEBUG "resultType": "vector", 23:02:06 DEBUG "result": [ 23:02:06 DEBUG { 23:02:06 DEBUG "metric": { 23:02:06 DEBUG "instance": "10.142.0.100:9100" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG [...] 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG "instance": "10.142.0.99:9100" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 1686693726, 23:02:06 DEBUG "0.00030000000000048505" 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG } 23:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693726" 23:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693726" 23:02:06 INFO [http_cmd]: http status code OK 23:02:06 DEBUG --- status code --- 23:02:06 DEBUG http status code is 200 (expected 200) 23:02:06 DEBUG --- http response --- 23:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:06 DEBUG ----- output ----- 23:02:06 DEBUG { 23:02:06 DEBUG "status": "success", 23:02:06 DEBUG "data": { 23:02:06 DEBUG "resultType": "vector", 23:02:06 DEBUG "result": [ 23:02:06 DEBUG { 23:02:06 DEBUG "metric": { 23:02:06 DEBUG "instance": "10.142.0.100:9100" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG [...] 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG "instance": "10.142.0.99:9100" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 1686693726, 23:02:06 DEBUG "20129950.8" 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG } 23:02:06 INFO [http_cmd]: http status code OK 23:02:06 DEBUG --- status code --- 23:02:06 DEBUG http status code is 200 (expected 200) 23:02:06 DEBUG --- http response --- 23:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:06 DEBUG ----- output ----- 23:02:06 DEBUG { 23:02:06 DEBUG "status": "success", 23:02:06 DEBUG "data": { 23:02:06 DEBUG "resultType": "vector", 23:02:06 DEBUG "result": [ 23:02:06 DEBUG { 23:02:06 DEBUG "metric": { 23:02:06 DEBUG "instance": "10.142.0.100:9100" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG [...] 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG "instance": "10.142.0.99:9100" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 1686693726, 23:02:06 DEBUG "20127543.4" 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG } 23:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693726" 23:02:06 INFO [http_cmd]: http status code OK 23:02:06 DEBUG --- status code --- 23:02:06 DEBUG http status code is 200 (expected 200) 23:02:06 DEBUG --- http response --- 23:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:06 DEBUG ----- output ----- 23:02:06 DEBUG { 23:02:06 DEBUG "status": "success", 23:02:06 DEBUG "data": { 23:02:06 DEBUG "resultType": "vector", 23:02:06 DEBUG "result": [ 23:02:06 DEBUG { 23:02:06 DEBUG "metric": { 23:02:06 DEBUG "pod": "ds-idrepo-2" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG [...] 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG "pod": "ds-idrepo-0" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 1686693726, 23:02:06 DEBUG "660.3" 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG } 23:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693726" 23:02:06 INFO [http_cmd]: http status code OK 23:02:06 DEBUG --- status code --- 23:02:06 DEBUG http status code is 200 (expected 200) 23:02:06 DEBUG --- http response --- 23:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:02:06 DEBUG ----- output ----- 23:02:06 DEBUG { 23:02:06 DEBUG "status": "success", 23:02:06 DEBUG "data": { 23:02:06 DEBUG "resultType": "vector", 23:02:06 DEBUG "result": [ 23:02:06 DEBUG { 23:02:06 DEBUG "metric": { 23:02:06 DEBUG "pod": "ds-cts-2" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG [...] 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 1686693726, 23:02:06 DEBUG "0" 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG } 23:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693726" 23:02:06 INFO [http_cmd]: http status code OK 23:02:06 DEBUG --- status code --- 23:02:06 DEBUG http status code is 200 (expected 200) 23:02:06 DEBUG --- http response --- 23:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:06 DEBUG ----- output ----- 23:02:06 DEBUG { 23:02:06 DEBUG "status": "success", 23:02:06 DEBUG "data": { 23:02:06 DEBUG "resultType": "vector", 23:02:06 DEBUG "result": [ 23:02:06 DEBUG { 23:02:06 DEBUG "metric": { 23:02:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG [...] 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG 23:02:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:02:06 DEBUG }, 23:02:06 DEBUG "value": [ 23:02:06 DEBUG 1686693726, 23:02:06 DEBUG "0.255265516047933" 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG ] 23:02:06 DEBUG } 23:02:06 DEBUG } 23:02:24 INFO 23:02:24 INFO [loop_until]: kubectl --namespace=xlou top pods 23:02:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:02:24 INFO [loop_until]: OK (rc = 0) 23:02:24 DEBUG --- stdout --- 23:02:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 9m 5712Mi am-7849cf7bdb-xrngh 6m 5690Mi ds-cts-0 10m 417Mi ds-cts-1 10m 425Mi ds-cts-2 10m 393Mi ds-idrepo-0 3382m 13804Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2980m 5003Mi idm-58dc667486-rgz9j 3005m 5102Mi lodemon-6455d5ff7c-zzndq 3m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 222m 644Mi 23:02:24 DEBUG --- stderr --- 23:02:24 DEBUG 23:02:28 INFO 23:02:28 INFO [loop_until]: kubectl --namespace=xlou top node 23:02:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:02:28 INFO [loop_until]: OK (rc = 0) 23:02:28 DEBUG --- stdout --- 23:02:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 2973m 18% 5941Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6864Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6828Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2815m 17% 6081Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 67m 0% 6954Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 438m 2% 3143Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1171Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1201Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13726Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3437m 21% 14533Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14618Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 289m 1% 1966Mi 3% 23:02:28 DEBUG --- stderr --- 23:02:28 DEBUG 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "0" 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG { 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "data": { 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG "result": [ 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG { 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "pod": "ds-idrepo-2" 23:03:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:03:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:03:05 DEBUG }, 23:03:05 DEBUG }, 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG "value": [ 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG [...] 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:03:05 DEBUG "pod": "ds-idrepo-0" 23:03:05 DEBUG 23:03:05 DEBUG }, 23:03:05 DEBUG }, 23:03:05 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 23:03:05 DEBUG "value": [ 23:03:05 DEBUG "value": [ 23:03:05 DEBUG }, 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG "0.2" 23:03:05 DEBUG "573.2333333333333" 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG ] 23:03:05 DEBUG ] 23:03:05 DEBUG "0" 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG ] 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG } 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "pod": "ds-idrepo-0" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "0" 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "0" 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "pod": "ds-cts-0" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "pod": "ds-idrepo-0" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "573.2333333333333" 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693785" 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "instance": "10.142.0.100:9100" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "instance": "10.142.0.99:9100" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "0" 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "0" 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "pod": "ds-idrepo-0" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "0" 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "pod": "ds-cts-0" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "pod": "ds-idrepo-0" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "0" 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693785" 23:03:05 INFO [http_cmd]: http status code OK 23:03:05 DEBUG --- status code --- 23:03:05 DEBUG http status code is 200 (expected 200) 23:03:05 DEBUG --- http response --- 23:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:03:05 DEBUG ----- output ----- 23:03:05 DEBUG { 23:03:05 DEBUG "status": "success", 23:03:05 DEBUG "data": { 23:03:05 DEBUG "resultType": "vector", 23:03:05 DEBUG "result": [ 23:03:05 DEBUG { 23:03:05 DEBUG "metric": { 23:03:05 DEBUG "pod": "ds-cts-0" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG [...] 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG 23:03:05 DEBUG "pod": "ds-cts-2" 23:03:05 DEBUG }, 23:03:05 DEBUG "value": [ 23:03:05 DEBUG 1686693785, 23:03:05 DEBUG "0" 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG ] 23:03:05 DEBUG } 23:03:05 DEBUG } 23:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693786" 23:03:06 INFO [http_cmd]: http status code OK 23:03:06 DEBUG --- status code --- 23:03:06 DEBUG http status code is 200 (expected 200) 23:03:06 DEBUG --- http response --- 23:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:06 DEBUG ----- output ----- 23:03:06 DEBUG { 23:03:06 DEBUG "status": "success", 23:03:06 DEBUG "data": { 23:03:06 DEBUG "resultType": "vector", 23:03:06 DEBUG "result": [ 23:03:06 DEBUG { 23:03:06 DEBUG "metric": { 23:03:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG [...] 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 1686693786, 23:03:06 DEBUG "0" 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG } 23:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693786" 23:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693786" 23:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693786" 23:03:06 INFO [http_cmd]: http status code OK 23:03:06 INFO [http_cmd]: http status code OK 23:03:06 DEBUG --- status code --- 23:03:06 DEBUG --- status code --- 23:03:06 DEBUG http status code is 200 (expected 200) 23:03:06 DEBUG http status code is 200 (expected 200) 23:03:06 DEBUG --- http response --- 23:03:06 DEBUG --- http response --- 23:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:06 DEBUG ----- output ----- 23:03:06 DEBUG { 23:03:06 DEBUG ----- output ----- 23:03:06 DEBUG "status": "success", 23:03:06 DEBUG { 23:03:06 DEBUG "data": { 23:03:06 DEBUG "status": "success", 23:03:06 DEBUG "resultType": "vector", 23:03:06 DEBUG "data": { 23:03:06 DEBUG "result": [ 23:03:06 DEBUG "resultType": "vector", 23:03:06 DEBUG { 23:03:06 DEBUG "result": [ 23:03:06 DEBUG "metric": { 23:03:06 DEBUG { 23:03:06 DEBUG "pod": "ds-idrepo-2" 23:03:06 DEBUG "metric": { 23:03:06 DEBUG }, 23:03:06 DEBUG "instance": "10.142.0.100:9100" 23:03:06 DEBUG "value": [ 23:03:06 DEBUG }, 23:03:06 DEBUG 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG [...] 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG [...] 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG "pod": "ds-idrepo-0" 23:03:06 DEBUG 23:03:06 DEBUG }, 23:03:06 DEBUG "instance": "10.142.0.99:9100" 23:03:06 DEBUG "value": [ 23:03:06 DEBUG }, 23:03:06 DEBUG 1686693786, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG "573.2333333333333" 23:03:06 DEBUG 1686693786, 23:03:06 DEBUG ] 23:03:06 INFO [http_cmd]: http status code OK 23:03:06 DEBUG "0.00023333333333350008" 23:03:06 DEBUG } 23:03:06 DEBUG --- status code --- 23:03:06 DEBUG ] 23:03:06 DEBUG ] 23:03:06 DEBUG http status code is 200 (expected 200) 23:03:06 DEBUG } 23:03:06 DEBUG } 23:03:06 DEBUG --- http response --- 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:06 DEBUG } 23:03:06 DEBUG ----- output ----- 23:03:06 DEBUG } 23:03:06 DEBUG { 23:03:06 DEBUG "status": "success", 23:03:06 DEBUG "data": { 23:03:06 DEBUG "resultType": "vector", 23:03:06 DEBUG "result": [ 23:03:06 DEBUG { 23:03:06 DEBUG "metric": { 23:03:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG [...] 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 1686693786, 23:03:06 DEBUG "13615804416" 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG } 23:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693786" 23:03:06 INFO [http_cmd]: http status code OK 23:03:06 DEBUG --- status code --- 23:03:06 DEBUG http status code is 200 (expected 200) 23:03:06 DEBUG --- http response --- 23:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:06 DEBUG ----- output ----- 23:03:06 DEBUG { 23:03:06 DEBUG "status": "success", 23:03:06 DEBUG "data": { 23:03:06 DEBUG "resultType": "vector", 23:03:06 DEBUG "result": [ 23:03:06 DEBUG { 23:03:06 DEBUG "metric": { 23:03:06 DEBUG "instance": "10.142.0.100:9100" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG [...] 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG "instance": "10.142.0.99:9100" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 1686693786, 23:03:06 DEBUG "21429182.8" 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG } 23:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693786" 23:03:06 INFO [http_cmd]: http status code OK 23:03:06 DEBUG --- status code --- 23:03:06 DEBUG http status code is 200 (expected 200) 23:03:06 DEBUG --- http response --- 23:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:06 DEBUG ----- output ----- 23:03:06 DEBUG { 23:03:06 DEBUG "status": "success", 23:03:06 DEBUG "data": { 23:03:06 DEBUG "resultType": "vector", 23:03:06 DEBUG "result": [ 23:03:06 DEBUG { 23:03:06 DEBUG "metric": { 23:03:06 DEBUG "instance": "10.142.0.100:9100" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG [...] 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG "instance": "10.142.0.99:9100" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 1686693786, 23:03:06 DEBUG "21424991.733333334" 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG } 23:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693786" 23:03:06 INFO [http_cmd]: http status code OK 23:03:06 DEBUG --- status code --- 23:03:06 DEBUG http status code is 200 (expected 200) 23:03:06 DEBUG --- http response --- 23:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:06 DEBUG ----- output ----- 23:03:06 DEBUG { 23:03:06 DEBUG "status": "success", 23:03:06 DEBUG "data": { 23:03:06 DEBUG "resultType": "vector", 23:03:06 DEBUG "result": [ 23:03:06 DEBUG { 23:03:06 DEBUG "metric": { 23:03:06 DEBUG "pod": "ds-idrepo-2" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG [...] 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG "pod": "ds-idrepo-0" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 1686693786, 23:03:06 DEBUG "573.2333333333333" 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG } 23:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693786" 23:03:06 INFO [http_cmd]: http status code OK 23:03:06 DEBUG --- status code --- 23:03:06 DEBUG http status code is 200 (expected 200) 23:03:06 DEBUG --- http response --- 23:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 23:03:06 DEBUG ----- output ----- 23:03:06 DEBUG { 23:03:06 DEBUG "status": "success", 23:03:06 DEBUG "data": { 23:03:06 DEBUG "resultType": "vector", 23:03:06 DEBUG "result": [ 23:03:06 DEBUG { 23:03:06 DEBUG "metric": { 23:03:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG [...] 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG "pod": "ds-cts-0" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 1686693786, 23:03:06 DEBUG "0.06988364373318424" 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG } 23:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693786" 23:03:06 INFO [http_cmd]: http status code OK 23:03:06 DEBUG --- status code --- 23:03:06 DEBUG http status code is 200 (expected 200) 23:03:06 DEBUG --- http response --- 23:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:06 DEBUG ----- output ----- 23:03:06 DEBUG { 23:03:06 DEBUG "status": "success", 23:03:06 DEBUG "data": { 23:03:06 DEBUG "resultType": "vector", 23:03:06 DEBUG "result": [ 23:03:06 DEBUG { 23:03:06 DEBUG "metric": { 23:03:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG [...] 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG 23:03:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:03:06 DEBUG }, 23:03:06 DEBUG "value": [ 23:03:06 DEBUG 1686693786, 23:03:06 DEBUG "0.23457136729391112" 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG ] 23:03:06 DEBUG } 23:03:06 DEBUG } 23:03:24 INFO 23:03:24 INFO [loop_until]: kubectl --namespace=xlou top pods 23:03:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:03:24 INFO [loop_until]: OK (rc = 0) 23:03:24 DEBUG --- stdout --- 23:03:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 19m 5722Mi am-7849cf7bdb-xrngh 6m 5690Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3375m 13799Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 14m 12911Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3347m 5005Mi idm-58dc667486-rgz9j 2515m 5102Mi lodemon-6455d5ff7c-zzndq 1m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 222m 645Mi 23:03:24 DEBUG --- stderr --- 23:03:24 DEBUG 23:03:28 INFO 23:03:28 INFO [loop_until]: kubectl --namespace=xlou top node 23:03:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:03:29 INFO [loop_until]: OK (rc = 0) 23:03:29 DEBUG --- stdout --- 23:03:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3406m 21% 5942Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 64m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6837Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2743m 17% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6955Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 440m 2% 3148Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 78m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 71m 0% 13731Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3334m 20% 14510Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14614Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 287m 1% 1968Mi 3% 23:03:29 DEBUG --- stderr --- 23:03:29 DEBUG 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG { 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "data": { 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "result": [ 23:04:05 DEBUG { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "pod": "ds-idrepo-2" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG [...] 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG "pod": "ds-idrepo-0" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG "706.1666666666666" 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG { 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "data": { 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "result": [ 23:04:05 DEBUG { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG [...] 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG "0" 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG { 23:04:05 DEBUG { 23:04:05 DEBUG { 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "data": { 23:04:05 DEBUG "data": { 23:04:05 DEBUG "data": { 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "result": [ 23:04:05 DEBUG "result": [ 23:04:05 DEBUG "result": [ 23:04:05 DEBUG { 23:04:05 DEBUG { 23:04:05 DEBUG { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:04:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:04:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:04:05 DEBUG }, 23:04:05 DEBUG }, 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG "value": [ 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG [...] 23:04:05 DEBUG [...] 23:04:05 DEBUG [...] 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG "pod": "ds-idrepo-0" 23:04:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:04:05 DEBUG "pod": "ds-idrepo-2" 23:04:05 DEBUG }, 23:04:05 DEBUG }, 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG "value": [ 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG "0" 23:04:05 DEBUG "0.2" 23:04:05 DEBUG "0" 23:04:05 DEBUG ] 23:04:05 DEBUG ] 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 DEBUG ] 23:04:05 DEBUG ] 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG { 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "data": { 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "result": [ 23:04:05 DEBUG { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG [...] 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG "0" 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG { 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "data": { 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "result": [ 23:04:05 DEBUG { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "pod": "ds-cts-0" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG [...] 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG "pod": "ds-idrepo-0" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG "706.1666666666666" 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693845" 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG { 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "data": { 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "result": [ 23:04:05 DEBUG { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "instance": "10.142.0.100:9100" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG [...] 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG "instance": "10.142.0.99:9100" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG "0" 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG { 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "data": { 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "result": [ 23:04:05 DEBUG { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG [...] 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG "0" 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG { 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "data": { 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "result": [ 23:04:05 DEBUG { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG [...] 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG "pod": "ds-idrepo-0" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG "0" 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693845" 23:04:05 INFO [http_cmd]: http status code OK 23:04:05 DEBUG --- status code --- 23:04:05 DEBUG http status code is 200 (expected 200) 23:04:05 DEBUG --- http response --- 23:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:05 DEBUG ----- output ----- 23:04:05 DEBUG { 23:04:05 DEBUG "status": "success", 23:04:05 DEBUG "data": { 23:04:05 DEBUG "resultType": "vector", 23:04:05 DEBUG "result": [ 23:04:05 DEBUG { 23:04:05 DEBUG "metric": { 23:04:05 DEBUG "pod": "ds-cts-0" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG [...] 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG 23:04:05 DEBUG "pod": "ds-idrepo-0" 23:04:05 DEBUG }, 23:04:05 DEBUG "value": [ 23:04:05 DEBUG 1686693845, 23:04:05 DEBUG "0" 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG ] 23:04:05 DEBUG } 23:04:05 DEBUG } 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693846" 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG { 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "data": { 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "result": [ 23:04:06 DEBUG { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "pod": "ds-cts-0" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG [...] 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG "pod": "ds-cts-2" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG "0" 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693846" 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG { 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "data": { 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "result": [ 23:04:06 DEBUG { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG [...] 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG "0" 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693846" 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG { 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "data": { 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "result": [ 23:04:06 DEBUG { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG [...] 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG "13616685056" 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693846" 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693846" 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG { 23:04:06 DEBUG { 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "data": { 23:04:06 DEBUG "data": { 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "result": [ 23:04:06 DEBUG "result": [ 23:04:06 DEBUG { 23:04:06 DEBUG { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "pod": "ds-idrepo-2" 23:04:06 DEBUG "instance": "10.142.0.100:9100" 23:04:06 DEBUG }, 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG [...] 23:04:06 DEBUG [...] 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG "pod": "ds-idrepo-0" 23:04:06 DEBUG "instance": "10.142.0.99:9100" 23:04:06 DEBUG }, 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG "706.1" 23:04:06 DEBUG "0.00036666666666747006" 23:04:06 DEBUG ] 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 DEBUG ] 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693846" 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG { 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "data": { 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "result": [ 23:04:06 DEBUG { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "instance": "10.142.0.100:9100" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG [...] 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG "instance": "10.142.0.99:9100" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG "22177164.366666667" 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693846" 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG { 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "data": { 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "result": [ 23:04:06 DEBUG { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "instance": "10.142.0.100:9100" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG [...] 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG "instance": "10.142.0.99:9100" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG "22172995.43333333" 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693846" 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG { 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "data": { 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "result": [ 23:04:06 DEBUG { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "pod": "ds-idrepo-2" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG [...] 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG "pod": "ds-idrepo-0" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG "706.1" 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693846" 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG { 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "data": { 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "result": [ 23:04:06 DEBUG { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "pod": "ds-cts-2" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG [...] 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG "0" 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693846" 23:04:06 INFO [http_cmd]: http status code OK 23:04:06 DEBUG --- status code --- 23:04:06 DEBUG http status code is 200 (expected 200) 23:04:06 DEBUG --- http response --- 23:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:06 DEBUG ----- output ----- 23:04:06 DEBUG { 23:04:06 DEBUG "status": "success", 23:04:06 DEBUG "data": { 23:04:06 DEBUG "resultType": "vector", 23:04:06 DEBUG "result": [ 23:04:06 DEBUG { 23:04:06 DEBUG "metric": { 23:04:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG [...] 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG 23:04:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:04:06 DEBUG }, 23:04:06 DEBUG "value": [ 23:04:06 DEBUG 1686693846, 23:04:06 DEBUG "0.24946592409915014" 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG ] 23:04:06 DEBUG } 23:04:06 DEBUG } 23:04:24 INFO 23:04:24 INFO [loop_until]: kubectl --namespace=xlou top pods 23:04:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:04:24 INFO [loop_until]: OK (rc = 0) 23:04:24 DEBUG --- stdout --- 23:04:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5761Mi am-7849cf7bdb-ntqzs 6m 5722Mi am-7849cf7bdb-xrngh 6m 5690Mi ds-cts-0 9m 418Mi ds-cts-1 8m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3514m 13803Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3554m 5008Mi idm-58dc667486-rgz9j 2720m 5102Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 235m 645Mi 23:04:24 DEBUG --- stderr --- 23:04:24 DEBUG 23:04:29 INFO 23:04:29 INFO [loop_until]: kubectl --namespace=xlou top node 23:04:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:04:29 INFO [loop_until]: OK (rc = 0) 23:04:29 DEBUG --- stdout --- 23:04:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3551m 22% 5943Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 70m 0% 6860Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 71m 0% 6840Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2950m 18% 6092Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6952Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 457m 2% 3140Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1480Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13731Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3690m 23% 14531Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 60m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14615Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 299m 1% 1966Mi 3% 23:04:29 DEBUG --- stderr --- 23:04:29 DEBUG 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 DEBUG --- status code --- 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:05 DEBUG { 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG { 23:05:05 DEBUG { 23:05:05 DEBUG "data": { 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG "data": { 23:05:05 DEBUG "data": { 23:05:05 DEBUG "result": [ 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG { 23:05:05 DEBUG "result": [ 23:05:05 DEBUG "result": [ 23:05:05 DEBUG "metric": { 23:05:05 DEBUG { 23:05:05 DEBUG { 23:05:05 DEBUG "pod": "ds-idrepo-2" 23:05:05 DEBUG "metric": { 23:05:05 DEBUG "metric": { 23:05:05 DEBUG }, 23:05:05 DEBUG "pod": "ds-cts-2" 23:05:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:05:05 DEBUG "value": [ 23:05:05 DEBUG }, 23:05:05 DEBUG }, 23:05:05 DEBUG 23:05:05 DEBUG "value": [ 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 DEBUG 23:05:05 DEBUG [...] 23:05:05 DEBUG 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG [...] 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG [...] 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 DEBUG "pod": "ds-idrepo-0" 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG }, 23:05:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:05:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:05:05 DEBUG { 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG "value": [ 23:05:05 DEBUG }, 23:05:05 DEBUG }, 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG "value": [ 23:05:05 DEBUG "data": { 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:05 DEBUG "568.7666666666667" 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG ] 23:05:05 DEBUG "0" 23:05:05 DEBUG "0.2" 23:05:05 DEBUG "result": [ 23:05:05 DEBUG { 23:05:05 DEBUG } 23:05:05 DEBUG ] 23:05:05 DEBUG ] 23:05:05 DEBUG { 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 DEBUG "metric": { 23:05:05 DEBUG "data": { 23:05:05 DEBUG } 23:05:05 DEBUG ] 23:05:05 DEBUG ] 23:05:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 DEBUG }, 23:05:05 DEBUG "result": [ 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 DEBUG "value": [ 23:05:05 DEBUG { 23:05:05 DEBUG 23:05:05 DEBUG "metric": { 23:05:05 DEBUG 23:05:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:05:05 DEBUG 23:05:05 DEBUG }, 23:05:05 DEBUG [...] 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:05:05 DEBUG [...] 23:05:05 DEBUG }, 23:05:05 DEBUG 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG 23:05:05 DEBUG "0" 23:05:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:05:05 DEBUG ] 23:05:05 DEBUG }, 23:05:05 DEBUG } 23:05:05 DEBUG "value": [ 23:05:05 DEBUG ] 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG } 23:05:05 DEBUG "0" 23:05:05 DEBUG } 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG { 23:05:05 DEBUG { 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG "data": { 23:05:05 DEBUG "data": { 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG "result": [ 23:05:05 DEBUG "result": [ 23:05:05 DEBUG { 23:05:05 DEBUG { 23:05:05 DEBUG "metric": { 23:05:05 DEBUG "metric": { 23:05:05 DEBUG "pod": "ds-cts-0" 23:05:05 DEBUG "pod": "ds-cts-2" 23:05:05 DEBUG }, 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG [...] 23:05:05 DEBUG [...] 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG "pod": "ds-idrepo-0" 23:05:05 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 23:05:05 DEBUG }, 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG "568.7333333333333" 23:05:05 DEBUG "0" 23:05:05 DEBUG ] 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 DEBUG ] 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG { 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG "data": { 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG "result": [ 23:05:05 DEBUG { 23:05:05 DEBUG "metric": { 23:05:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG [...] 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG "0" 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693905" 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG { 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG "data": { 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG "result": [ 23:05:05 DEBUG { 23:05:05 DEBUG "metric": { 23:05:05 DEBUG "instance": "10.142.0.100:9100" 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG [...] 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG "instance": "10.142.0.99:9100" 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG "0" 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG { 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG "data": { 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG "result": [ 23:05:05 DEBUG { 23:05:05 DEBUG "metric": { 23:05:05 DEBUG "pod": "ds-cts-2" 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG [...] 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG "0" 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693905" 23:05:05 INFO [http_cmd]: http status code OK 23:05:05 DEBUG --- status code --- 23:05:05 DEBUG http status code is 200 (expected 200) 23:05:05 DEBUG --- http response --- 23:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:05 DEBUG ----- output ----- 23:05:05 DEBUG { 23:05:05 DEBUG "status": "success", 23:05:05 DEBUG "data": { 23:05:05 DEBUG "resultType": "vector", 23:05:05 DEBUG "result": [ 23:05:05 DEBUG { 23:05:05 DEBUG "metric": { 23:05:05 DEBUG "pod": "ds-cts-0" 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG [...] 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG 23:05:05 DEBUG "pod": "ds-idrepo-0" 23:05:05 DEBUG }, 23:05:05 DEBUG "value": [ 23:05:05 DEBUG 1686693905, 23:05:05 DEBUG "0" 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG ] 23:05:05 DEBUG } 23:05:05 DEBUG } 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693906" 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG --- status code --- 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG { 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG "data": { 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG "result": [ 23:05:06 DEBUG { 23:05:06 DEBUG "metric": { 23:05:06 DEBUG "pod": "ds-cts-0" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG "pod": "ds-cts-2" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG "0" 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693906" 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG --- status code --- 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG { 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG "data": { 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG "result": [ 23:05:06 DEBUG { 23:05:06 DEBUG "metric": { 23:05:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG "0" 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693906" 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693906" 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG --- status code --- 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG --- status code --- 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:06 DEBUG { 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG { 23:05:06 DEBUG "data": { 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG "data": { 23:05:06 DEBUG "result": [ 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG { 23:05:06 DEBUG "result": [ 23:05:06 DEBUG "metric": { 23:05:06 DEBUG { 23:05:06 DEBUG "instance": "10.142.0.100:9100" 23:05:06 DEBUG "metric": { 23:05:06 DEBUG }, 23:05:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:05:06 DEBUG "value": [ 23:05:06 DEBUG }, 23:05:06 DEBUG 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG "instance": "10.142.0.99:9100" 23:05:06 DEBUG 23:05:06 DEBUG }, 23:05:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:05:06 DEBUG "value": [ 23:05:06 DEBUG }, 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG "0.0003333333333330302" 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG ] 23:05:06 DEBUG "13617037312" 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693906" 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG --- status code --- 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG { 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG "data": { 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG "result": [ 23:05:06 DEBUG { 23:05:06 DEBUG "metric": { 23:05:06 DEBUG "pod": "ds-idrepo-2" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG "pod": "ds-idrepo-0" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG "568.7666666666667" 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693906" 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG --- status code --- 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG { 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG "data": { 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG "result": [ 23:05:06 DEBUG { 23:05:06 DEBUG "metric": { 23:05:06 DEBUG "instance": "10.142.0.100:9100" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG "instance": "10.142.0.99:9100" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG "18719970.833333332" 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693906" 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG --- status code --- 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG { 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG "data": { 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG "result": [ 23:05:06 DEBUG { 23:05:06 DEBUG "metric": { 23:05:06 DEBUG "instance": "10.142.0.100:9100" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG "instance": "10.142.0.99:9100" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG "18715949.633333333" 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693906" 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG --- status code --- 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG { 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG "data": { 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG "result": [ 23:05:06 DEBUG { 23:05:06 DEBUG "metric": { 23:05:06 DEBUG "pod": "ds-idrepo-2" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG "pod": "ds-idrepo-0" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG "568.8" 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693906" 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG --- status code --- 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG { 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG "data": { 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG "result": [ 23:05:06 DEBUG { 23:05:06 DEBUG "metric": { 23:05:06 DEBUG "pod": "ds-cts-2" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG "pod": "end-user-ui-5c589c665b-7tbkq" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG "0" 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693906" 23:05:06 INFO [http_cmd]: http status code OK 23:05:06 DEBUG --- status code --- 23:05:06 DEBUG http status code is 200 (expected 200) 23:05:06 DEBUG --- http response --- 23:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:06 DEBUG ----- output ----- 23:05:06 DEBUG { 23:05:06 DEBUG "status": "success", 23:05:06 DEBUG "data": { 23:05:06 DEBUG "resultType": "vector", 23:05:06 DEBUG "result": [ 23:05:06 DEBUG { 23:05:06 DEBUG "metric": { 23:05:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG [...] 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG 23:05:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:05:06 DEBUG }, 23:05:06 DEBUG "value": [ 23:05:06 DEBUG 1686693906, 23:05:06 DEBUG "0.2477615710998757" 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG ] 23:05:06 DEBUG } 23:05:06 DEBUG } 23:05:25 INFO 23:05:25 INFO [loop_until]: kubectl --namespace=xlou top pods 23:05:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:05:25 INFO [loop_until]: OK (rc = 0) 23:05:25 DEBUG --- stdout --- 23:05:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 7m 5760Mi am-7849cf7bdb-ntqzs 6m 5722Mi am-7849cf7bdb-xrngh 6m 5690Mi ds-cts-0 9m 417Mi ds-cts-1 10m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3163m 13810Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3077m 5009Mi idm-58dc667486-rgz9j 2673m 5102Mi lodemon-6455d5ff7c-zzndq 4m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 208m 644Mi 23:05:25 DEBUG --- stderr --- 23:05:25 DEBUG 23:05:29 INFO 23:05:29 INFO [loop_until]: kubectl --namespace=xlou top node 23:05:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:05:29 INFO [loop_until]: OK (rc = 0) 23:05:29 DEBUG --- stdout --- 23:05:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3199m 20% 5942Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 67m 0% 6861Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 69m 0% 6839Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2638m 16% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 69m 0% 6953Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 417m 2% 3150Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 79m 0% 1476Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1201Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 68m 0% 13734Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3165m 19% 14520Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 70m 0% 14615Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 279m 1% 1965Mi 3% 23:05:29 DEBUG --- stderr --- 23:05:29 DEBUG 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693965" 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693965" 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693965" 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693965" 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693965" 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693965" 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693965" 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693965" 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 DEBUG --- status code --- 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG --- status code --- 23:06:05 DEBUG --- status code --- 23:06:05 DEBUG --- status code --- 23:06:05 DEBUG --- status code --- 23:06:05 DEBUG --- status code --- 23:06:05 DEBUG --- status code --- 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG { 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG { 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG { 23:06:05 DEBUG { 23:06:05 DEBUG { 23:06:05 DEBUG { 23:06:05 DEBUG { 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG "data": { 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG "data": { 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "data": { 23:06:05 DEBUG "data": { 23:06:05 DEBUG "data": { 23:06:05 DEBUG "data": { 23:06:05 DEBUG "data": { 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "result": [ 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "result": [ 23:06:05 DEBUG { 23:06:05 DEBUG "result": [ 23:06:05 DEBUG "result": [ 23:06:05 DEBUG "result": [ 23:06:05 DEBUG "result": [ 23:06:05 DEBUG "result": [ 23:06:05 DEBUG { 23:06:05 DEBUG "metric": { 23:06:05 DEBUG { 23:06:05 DEBUG { 23:06:05 DEBUG { 23:06:05 DEBUG { 23:06:05 DEBUG { 23:06:05 DEBUG "metric": { 23:06:05 DEBUG "pod": "ds-cts-0" 23:06:05 DEBUG "metric": { 23:06:05 DEBUG "metric": { 23:06:05 DEBUG "metric": { 23:06:05 DEBUG "metric": { 23:06:05 DEBUG "metric": { 23:06:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:06:05 DEBUG }, 23:06:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:06:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:06:05 DEBUG "pod": "ds-cts-2" 23:06:05 DEBUG "pod": "ds-cts-2" 23:06:05 DEBUG "pod": "ds-idrepo-2" 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG }, 23:06:05 DEBUG }, 23:06:05 DEBUG }, 23:06:05 DEBUG }, 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 23:06:05 DEBUG "value": [ 23:06:05 DEBUG "value": [ 23:06:05 DEBUG "value": [ 23:06:05 DEBUG "value": [ 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG [...] 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG [...] 23:06:05 DEBUG 23:06:05 DEBUG [...] 23:06:05 DEBUG [...] 23:06:05 DEBUG [...] 23:06:05 DEBUG [...] 23:06:05 DEBUG [...] 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG "pod": "ds-idrepo-0" 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:06:05 DEBUG }, 23:06:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:06:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:06:05 DEBUG "pod": "ds-idrepo-0" 23:06:05 DEBUG "pod": "ds-idrepo-0" 23:06:05 DEBUG "pod": "ds-idrepo-0" 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG }, 23:06:05 DEBUG }, 23:06:05 DEBUG }, 23:06:05 DEBUG }, 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG "value": [ 23:06:05 DEBUG "value": [ 23:06:05 DEBUG "value": [ 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG "631.6" 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG "0" 23:06:05 DEBUG ] 23:06:05 DEBUG "0" 23:06:05 DEBUG "0.2" 23:06:05 DEBUG "3.062650643579492" 23:06:05 DEBUG "6279.1819269681255" 23:06:05 DEBUG "631.6" 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG ] 23:06:05 DEBUG ] 23:06:05 DEBUG ] 23:06:05 DEBUG ] 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG ] 23:06:05 DEBUG ] 23:06:05 DEBUG ] 23:06:05 DEBUG ] 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 DEBUG --- status code --- 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG { 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG "data": { 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "result": [ 23:06:05 DEBUG { 23:06:05 DEBUG "metric": { 23:06:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG [...] 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG "0" 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693965" 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 DEBUG --- status code --- 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG { 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG "data": { 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "result": [ 23:06:05 DEBUG { 23:06:05 DEBUG "metric": { 23:06:05 DEBUG "instance": "10.142.0.100:9100" 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG [...] 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG "instance": "10.142.0.99:9100" 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG "0" 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693965" 23:06:05 INFO [http_cmd]: http status code OK 23:06:05 DEBUG --- status code --- 23:06:05 DEBUG http status code is 200 (expected 200) 23:06:05 DEBUG --- http response --- 23:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:05 DEBUG ----- output ----- 23:06:05 DEBUG { 23:06:05 DEBUG "status": "success", 23:06:05 DEBUG "data": { 23:06:05 DEBUG "resultType": "vector", 23:06:05 DEBUG "result": [ 23:06:05 DEBUG { 23:06:05 DEBUG "metric": { 23:06:05 DEBUG "pod": "ds-cts-2" 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG [...] 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG 23:06:05 DEBUG "pod": "ds-idrepo-0" 23:06:05 DEBUG }, 23:06:05 DEBUG "value": [ 23:06:05 DEBUG 1686693965, 23:06:05 DEBUG "545100112.1224209" 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG ] 23:06:05 DEBUG } 23:06:05 DEBUG } 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693966" 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG { 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG "data": { 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG "result": [ 23:06:06 DEBUG { 23:06:06 DEBUG "metric": { 23:06:06 DEBUG "pod": "ds-cts-0" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG [...] 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG "pod": "ds-idrepo-0" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG "0" 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG } 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693966" 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693966" 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG { 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG "data": { 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG "result": [ 23:06:06 DEBUG { 23:06:06 DEBUG { 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG "metric": { 23:06:06 DEBUG "data": { 23:06:06 DEBUG "pod": "ds-cts-0" 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG }, 23:06:06 DEBUG "result": [ 23:06:06 DEBUG "value": [ 23:06:06 DEBUG { 23:06:06 DEBUG 23:06:06 DEBUG "metric": { 23:06:06 DEBUG 23:06:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:06:06 DEBUG 23:06:06 DEBUG }, 23:06:06 DEBUG [...] 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG "pod": "ds-cts-2" 23:06:06 DEBUG [...] 23:06:06 DEBUG }, 23:06:06 DEBUG 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG 23:06:06 DEBUG "0" 23:06:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:06:06 DEBUG ] 23:06:06 DEBUG }, 23:06:06 DEBUG } 23:06:06 DEBUG "value": [ 23:06:06 DEBUG ] 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG } 23:06:06 DEBUG "0" 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG } 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693966" 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686693966" 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG { 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG "data": { 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG "result": [ 23:06:06 DEBUG { 23:06:06 DEBUG { 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG "metric": { 23:06:06 DEBUG "data": { 23:06:06 DEBUG "instance": "10.142.0.100:9100" 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG }, 23:06:06 DEBUG "result": [ 23:06:06 DEBUG "value": [ 23:06:06 DEBUG { 23:06:06 DEBUG 23:06:06 DEBUG "metric": { 23:06:06 DEBUG 23:06:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:06:06 DEBUG 23:06:06 DEBUG }, 23:06:06 DEBUG [...] 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG "instance": "10.142.0.99:9100" 23:06:06 DEBUG [...] 23:06:06 DEBUG }, 23:06:06 DEBUG 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG 23:06:06 DEBUG "0.00030000000000048505" 23:06:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:06:06 DEBUG ] 23:06:06 DEBUG }, 23:06:06 DEBUG } 23:06:06 DEBUG "value": [ 23:06:06 DEBUG ] 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG } 23:06:06 DEBUG "13618364416" 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG } 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693966" 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG { 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG "data": { 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG "result": [ 23:06:06 DEBUG { 23:06:06 DEBUG "metric": { 23:06:06 DEBUG "pod": "ds-idrepo-2" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG [...] 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG "pod": "ds-idrepo-0" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG "631.6" 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG } 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693966" 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG { 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG "data": { 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG "result": [ 23:06:06 DEBUG { 23:06:06 DEBUG "metric": { 23:06:06 DEBUG "instance": "10.142.0.100:9100" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG [...] 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG "instance": "10.142.0.99:9100" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG "20645499.566666666" 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG } 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686693966" 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG { 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG "data": { 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG "result": [ 23:06:06 DEBUG { 23:06:06 DEBUG "metric": { 23:06:06 DEBUG "instance": "10.142.0.100:9100" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG [...] 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG "instance": "10.142.0.99:9100" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG "20641561.1" 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG } 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693966" 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG { 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG "data": { 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG "result": [ 23:06:06 DEBUG { 23:06:06 DEBUG "metric": { 23:06:06 DEBUG "pod": "ds-idrepo-2" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG [...] 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG "pod": "ds-idrepo-0" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG "631.6" 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG } 23:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686693966" 23:06:06 INFO [http_cmd]: http status code OK 23:06:06 DEBUG --- status code --- 23:06:06 DEBUG http status code is 200 (expected 200) 23:06:06 DEBUG --- http response --- 23:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:06 DEBUG ----- output ----- 23:06:06 DEBUG { 23:06:06 DEBUG "status": "success", 23:06:06 DEBUG "data": { 23:06:06 DEBUG "resultType": "vector", 23:06:06 DEBUG "result": [ 23:06:06 DEBUG { 23:06:06 DEBUG "metric": { 23:06:06 DEBUG "pod": "ds-cts-2" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG [...] 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG 23:06:06 DEBUG "pod": "ds-idrepo-0" 23:06:06 DEBUG }, 23:06:06 DEBUG "value": [ 23:06:06 DEBUG 1686693966, 23:06:06 DEBUG "0.543931537150524" 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG ] 23:06:06 DEBUG } 23:06:06 DEBUG } 23:06:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686693967" 23:06:07 INFO [http_cmd]: http status code OK 23:06:07 DEBUG --- status code --- 23:06:07 DEBUG http status code is 200 (expected 200) 23:06:07 DEBUG --- http response --- 23:06:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:07 DEBUG ----- output ----- 23:06:07 DEBUG { 23:06:07 DEBUG "status": "success", 23:06:07 DEBUG "data": { 23:06:07 DEBUG "resultType": "vector", 23:06:07 DEBUG "result": [ 23:06:07 DEBUG { 23:06:07 DEBUG "metric": { 23:06:07 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:06:07 DEBUG }, 23:06:07 DEBUG "value": [ 23:06:07 DEBUG 23:06:07 DEBUG 23:06:07 DEBUG 23:06:07 DEBUG [...] 23:06:07 DEBUG 23:06:07 DEBUG 23:06:07 DEBUG 23:06:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:06:07 DEBUG }, 23:06:07 DEBUG "value": [ 23:06:07 DEBUG 1686693967, 23:06:07 DEBUG "0.23557303214064698" 23:06:07 DEBUG ] 23:06:07 DEBUG } 23:06:07 DEBUG ] 23:06:07 DEBUG } 23:06:07 DEBUG } 23:06:25 INFO 23:06:25 INFO [loop_until]: kubectl --namespace=xlou top pods 23:06:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:06:25 INFO [loop_until]: OK (rc = 0) 23:06:25 DEBUG --- stdout --- 23:06:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 9m 5760Mi am-7849cf7bdb-ntqzs 6m 5722Mi am-7849cf7bdb-xrngh 6m 5690Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 3497m 13807Mi ds-idrepo-1 16m 13820Mi ds-idrepo-2 16m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 3088m 5010Mi idm-58dc667486-rgz9j 2975m 5102Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 224m 645Mi 23:06:25 DEBUG --- stderr --- 23:06:25 DEBUG 23:06:29 INFO 23:06:29 INFO [loop_until]: kubectl --namespace=xlou top node 23:06:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:06:29 INFO [loop_until]: OK (rc = 0) 23:06:29 DEBUG --- stdout --- 23:06:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3390m 21% 5945Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6866Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6839Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 3259m 20% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 68m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 469m 2% 3127Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 82m 0% 1477Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 58m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13732Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3674m 23% 14512Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14614Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 291m 1% 1968Mi 3% 23:06:29 DEBUG --- stderr --- 23:06:29 DEBUG 23:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694025" 23:07:05 INFO [http_cmd]: http status code OK 23:07:05 DEBUG --- status code --- 23:07:05 DEBUG http status code is 200 (expected 200) 23:07:05 DEBUG --- http response --- 23:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:07:05 DEBUG ----- output ----- 23:07:05 DEBUG { 23:07:05 DEBUG "status": "success", 23:07:05 DEBUG "data": { 23:07:05 DEBUG "resultType": "vector", 23:07:05 DEBUG "result": [ 23:07:05 DEBUG { 23:07:05 DEBUG "metric": { 23:07:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:07:05 DEBUG }, 23:07:05 DEBUG "value": [ 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG [...] 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG "pod": "idm-58dc667486-rgz9j" 23:07:05 DEBUG }, 23:07:05 DEBUG "value": [ 23:07:05 DEBUG 1686694025, 23:07:05 DEBUG "0" 23:07:05 DEBUG ] 23:07:05 DEBUG } 23:07:05 DEBUG ] 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694025" 23:07:05 INFO [http_cmd]: http status code OK 23:07:05 DEBUG --- status code --- 23:07:05 DEBUG http status code is 200 (expected 200) 23:07:05 DEBUG --- http response --- 23:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:05 DEBUG ----- output ----- 23:07:05 DEBUG { 23:07:05 DEBUG "status": "success", 23:07:05 DEBUG "data": { 23:07:05 DEBUG "resultType": "vector", 23:07:05 DEBUG "result": [ 23:07:05 DEBUG { 23:07:05 DEBUG "metric": { 23:07:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:07:05 DEBUG }, 23:07:05 DEBUG "value": [ 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG [...] 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:07:05 DEBUG }, 23:07:05 DEBUG "value": [ 23:07:05 DEBUG 1686694025, 23:07:05 DEBUG "0" 23:07:05 DEBUG ] 23:07:05 DEBUG } 23:07:05 DEBUG ] 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694025" 23:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694025" 23:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694025" 23:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694025" 23:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694025" 23:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694025" 23:07:05 INFO [http_cmd]: http status code OK 23:07:05 INFO [http_cmd]: http status code OK 23:07:05 DEBUG --- status code --- 23:07:05 DEBUG --- status code --- 23:07:05 DEBUG http status code is 200 (expected 200) 23:07:05 DEBUG http status code is 200 (expected 200) 23:07:05 DEBUG --- http response --- 23:07:05 DEBUG --- http response --- 23:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:05 DEBUG ----- output ----- 23:07:05 DEBUG ----- output ----- 23:07:05 DEBUG { 23:07:05 DEBUG { 23:07:05 DEBUG "status": "success", 23:07:05 DEBUG "status": "success", 23:07:05 DEBUG "data": { 23:07:05 DEBUG "data": { 23:07:05 DEBUG "resultType": "vector", 23:07:05 DEBUG "resultType": "vector", 23:07:05 DEBUG "result": [ 23:07:05 DEBUG "result": [ 23:07:05 DEBUG { 23:07:05 DEBUG { 23:07:05 DEBUG "metric": { 23:07:05 DEBUG "metric": { 23:07:05 DEBUG "pod": "ds-cts-0" 23:07:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:07:05 DEBUG }, 23:07:05 DEBUG }, 23:07:05 DEBUG "value": [ 23:07:05 DEBUG "value": [ 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 INFO [http_cmd]: http status code OK 23:07:05 INFO [http_cmd]: http status code OK 23:07:05 DEBUG [...] 23:07:05 INFO [http_cmd]: http status code OK 23:07:05 INFO [http_cmd]: http status code OK 23:07:05 DEBUG [...] 23:07:05 DEBUG --- status code --- 23:07:05 DEBUG --- status code --- 23:07:05 DEBUG 23:07:05 DEBUG --- status code --- 23:07:05 DEBUG --- status code --- 23:07:05 DEBUG 23:07:05 DEBUG http status code is 200 (expected 200) 23:07:05 DEBUG http status code is 200 (expected 200) 23:07:05 DEBUG 23:07:05 DEBUG http status code is 200 (expected 200) 23:07:05 DEBUG http status code is 200 (expected 200) 23:07:05 DEBUG 23:07:05 DEBUG --- http response --- 23:07:05 DEBUG --- http response --- 23:07:05 DEBUG 23:07:05 DEBUG --- http response --- 23:07:05 DEBUG --- http response --- 23:07:05 DEBUG 23:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:05 DEBUG "pod": "ds-idrepo-0" 23:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:07:05 DEBUG ----- output ----- 23:07:05 DEBUG ----- output ----- 23:07:05 DEBUG }, 23:07:05 DEBUG ----- output ----- 23:07:05 DEBUG ----- output ----- 23:07:05 DEBUG }, 23:07:05 DEBUG { 23:07:05 DEBUG { 23:07:05 DEBUG "value": [ 23:07:05 DEBUG { 23:07:05 DEBUG { 23:07:05 DEBUG "value": [ 23:07:05 DEBUG "status": "success", 23:07:05 DEBUG "status": "success", 23:07:05 DEBUG 1686694025, 23:07:05 DEBUG "status": "success", 23:07:05 DEBUG "status": "success", 23:07:05 DEBUG 1686694025, 23:07:05 DEBUG "data": { 23:07:05 DEBUG "data": { 23:07:05 DEBUG "610.8666666666667" 23:07:05 DEBUG "data": { 23:07:05 DEBUG "data": { 23:07:05 DEBUG "0" 23:07:05 DEBUG "resultType": "vector", 23:07:05 DEBUG "resultType": "vector", 23:07:05 DEBUG ] 23:07:05 DEBUG "resultType": "vector", 23:07:05 DEBUG "resultType": "vector", 23:07:05 DEBUG ] 23:07:05 DEBUG "result": [ 23:07:05 DEBUG "result": [ 23:07:05 DEBUG } 23:07:05 DEBUG "result": [ 23:07:05 DEBUG "result": [ 23:07:05 DEBUG } 23:07:05 DEBUG { 23:07:05 DEBUG { 23:07:05 DEBUG ] 23:07:05 DEBUG { 23:07:05 DEBUG { 23:07:05 DEBUG ] 23:07:05 DEBUG "metric": { 23:07:05 DEBUG "metric": { 23:07:05 DEBUG } 23:07:05 DEBUG "metric": { 23:07:05 DEBUG "metric": { 23:07:05 DEBUG } 23:07:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:07:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:07:05 DEBUG } 23:07:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:07:05 DEBUG "pod": "ds-idrepo-2" 23:07:05 DEBUG } 23:07:05 DEBUG }, 23:07:05 DEBUG }, 23:07:05 DEBUG }, 23:07:05 DEBUG }, 23:07:05 DEBUG "value": [ 23:07:05 DEBUG "value": [ 23:07:05 DEBUG "value": [ 23:07:05 DEBUG "value": [ 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG [...] 23:07:05 DEBUG [...] 23:07:05 DEBUG [...] 23:07:05 DEBUG [...] 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG 23:07:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:07:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:07:05 DEBUG "pod": "idm-58dc667486-ql6zm" 23:07:05 DEBUG "pod": "ds-idrepo-0" 23:07:05 DEBUG }, 23:07:05 DEBUG }, 23:07:05 DEBUG }, 23:07:05 DEBUG }, 23:07:05 DEBUG "value": [ 23:07:05 DEBUG "value": [ 23:07:05 DEBUG "value": [ 23:07:05 DEBUG "value": [ 23:07:05 DEBUG 1686694025, 23:07:05 DEBUG 1686694025, 23:07:05 DEBUG 1686694025, 23:07:05 DEBUG 1686694025, 23:07:05 DEBUG "0" 23:07:05 DEBUG "0.2" 23:07:05 DEBUG "0" 23:07:05 DEBUG "610.8666666666667" 23:07:05 DEBUG ] 23:07:05 DEBUG ] 23:07:05 DEBUG ] 23:07:05 DEBUG ] 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 DEBUG ] 23:07:05 DEBUG ] 23:07:05 DEBUG ] 23:07:05 DEBUG ] 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:05 DEBUG } 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694026" 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "instance": "10.142.0.100:9100" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "instance": "10.142.0.99:9100" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "0" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694026" 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "pod": "ds-cts-2" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "pod": "idm-58dc667486-rgz9j" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "0" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694026" 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "pod": "ds-cts-0" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "pod": "ds-idrepo-0" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "0" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694026" 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694026" 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "pod": "ds-cts-0" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "pod": "ds-cts-2" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "0" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "0" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686694026" 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694026" 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "13617143808" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "instance": "10.142.0.100:9100" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "instance": "10.142.0.99:9100" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "0.0003333333333330302" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694026" 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "pod": "ds-idrepo-2" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "pod": "ds-idrepo-0" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "610.8333333333334" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694026" 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "instance": "10.142.0.100:9100" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "instance": "10.142.0.99:9100" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "19724327.366666667" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694026" 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "instance": "10.142.0.100:9100" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "instance": "10.142.0.99:9100" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "19722083.666666664" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694026" 23:07:06 INFO [http_cmd]: http status code OK 23:07:06 DEBUG --- status code --- 23:07:06 DEBUG http status code is 200 (expected 200) 23:07:06 DEBUG --- http response --- 23:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:06 DEBUG ----- output ----- 23:07:06 DEBUG { 23:07:06 DEBUG "status": "success", 23:07:06 DEBUG "data": { 23:07:06 DEBUG "resultType": "vector", 23:07:06 DEBUG "result": [ 23:07:06 DEBUG { 23:07:06 DEBUG "metric": { 23:07:06 DEBUG "pod": "ds-idrepo-2" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG [...] 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG 23:07:06 DEBUG "pod": "ds-idrepo-0" 23:07:06 DEBUG }, 23:07:06 DEBUG "value": [ 23:07:06 DEBUG 1686694026, 23:07:06 DEBUG "610.8333333333334" 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG ] 23:07:06 DEBUG } 23:07:06 DEBUG } 23:07:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694027" 23:07:07 INFO [http_cmd]: http status code OK 23:07:07 DEBUG --- status code --- 23:07:07 DEBUG http status code is 200 (expected 200) 23:07:07 DEBUG --- http response --- 23:07:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:07:07 DEBUG ----- output ----- 23:07:07 DEBUG { 23:07:07 DEBUG "status": "success", 23:07:07 DEBUG "data": { 23:07:07 DEBUG "resultType": "vector", 23:07:07 DEBUG "result": [ 23:07:07 DEBUG { 23:07:07 DEBUG "metric": { 23:07:07 DEBUG "pod": "ds-cts-2" 23:07:07 DEBUG }, 23:07:07 DEBUG "value": [ 23:07:07 DEBUG 23:07:07 DEBUG 23:07:07 DEBUG 23:07:07 DEBUG [...] 23:07:07 DEBUG 23:07:07 DEBUG 23:07:07 DEBUG 23:07:07 DEBUG "pod": "ds-idrepo-2" 23:07:07 DEBUG }, 23:07:07 DEBUG "value": [ 23:07:07 DEBUG 1686694027, 23:07:07 DEBUG "0.7116556824159163" 23:07:07 DEBUG ] 23:07:07 DEBUG } 23:07:07 DEBUG ] 23:07:07 DEBUG } 23:07:07 DEBUG } 23:07:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686694027" 23:07:07 INFO [http_cmd]: http status code OK 23:07:07 DEBUG --- status code --- 23:07:07 DEBUG http status code is 200 (expected 200) 23:07:07 DEBUG --- http response --- 23:07:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:07 DEBUG ----- output ----- 23:07:07 DEBUG { 23:07:07 DEBUG "status": "success", 23:07:07 DEBUG "data": { 23:07:07 DEBUG "resultType": "vector", 23:07:07 DEBUG "result": [ 23:07:07 DEBUG { 23:07:07 DEBUG "metric": { 23:07:07 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:07:07 DEBUG }, 23:07:07 DEBUG "value": [ 23:07:07 DEBUG 23:07:07 DEBUG 23:07:07 DEBUG 23:07:07 DEBUG [...] 23:07:07 DEBUG 23:07:07 DEBUG 23:07:07 DEBUG 23:07:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:07:07 DEBUG }, 23:07:07 DEBUG "value": [ 23:07:07 DEBUG 1686694027, 23:07:07 DEBUG "0.2434318093003182" 23:07:07 DEBUG ] 23:07:07 DEBUG } 23:07:07 DEBUG ] 23:07:07 DEBUG } 23:07:07 DEBUG } 23:07:25 INFO 23:07:25 INFO [loop_until]: kubectl --namespace=xlou top pods 23:07:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:07:25 INFO [loop_until]: OK (rc = 0) 23:07:25 DEBUG --- stdout --- 23:07:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 5m 5722Mi am-7849cf7bdb-xrngh 6m 5690Mi ds-cts-0 10m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3320m 13801Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 14m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2789m 5013Mi idm-58dc667486-rgz9j 2864m 5102Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 214m 644Mi 23:07:25 DEBUG --- stderr --- 23:07:25 DEBUG 23:07:29 INFO 23:07:29 INFO [loop_until]: kubectl --namespace=xlou top node 23:07:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:07:29 INFO [loop_until]: OK (rc = 0) 23:07:29 DEBUG --- stdout --- 23:07:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3002m 18% 5949Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 66m 0% 6864Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 68m 0% 6841Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 3206m 20% 6075Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 68m 0% 6953Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 441m 2% 3133Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 83m 0% 1479Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1162Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 61m 0% 1204Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 72m 0% 13731Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3494m 21% 14544Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 66m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14613Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 279m 1% 1968Mi 3% 23:07:29 DEBUG --- stderr --- 23:07:29 DEBUG 23:08:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694085" 23:08:05 INFO [http_cmd]: http status code OK 23:08:05 DEBUG --- status code --- 23:08:05 DEBUG http status code is 200 (expected 200) 23:08:05 DEBUG --- http response --- 23:08:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:08:05 DEBUG ----- output ----- 23:08:05 DEBUG { 23:08:05 DEBUG "status": "success", 23:08:05 DEBUG "data": { 23:08:05 DEBUG "resultType": "vector", 23:08:05 DEBUG "result": [ 23:08:05 DEBUG { 23:08:05 DEBUG "metric": { 23:08:05 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:08:05 DEBUG }, 23:08:05 DEBUG "value": [ 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG [...] 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG "pod": "ds-cts-0" 23:08:05 DEBUG }, 23:08:05 DEBUG "value": [ 23:08:05 DEBUG 1686694085, 23:08:05 DEBUG "0.008297183324621307" 23:08:05 DEBUG ] 23:08:05 DEBUG } 23:08:05 DEBUG ] 23:08:05 DEBUG } 23:08:05 DEBUG } 23:08:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694085" 23:08:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694085" 23:08:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694085" 23:08:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694085" 23:08:05 INFO [http_cmd]: http status code OK 23:08:05 DEBUG --- status code --- 23:08:05 DEBUG http status code is 200 (expected 200) 23:08:05 DEBUG --- http response --- 23:08:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:05 DEBUG ----- output ----- 23:08:05 DEBUG { 23:08:05 DEBUG "status": "success", 23:08:05 DEBUG "data": { 23:08:05 DEBUG "resultType": "vector", 23:08:05 DEBUG "result": [ 23:08:05 DEBUG { 23:08:05 DEBUG "metric": { 23:08:05 DEBUG "pod": "ds-cts-0" 23:08:05 DEBUG }, 23:08:05 DEBUG "value": [ 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG [...] 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG "pod": "ds-idrepo-0" 23:08:05 DEBUG }, 23:08:05 DEBUG "value": [ 23:08:05 DEBUG 1686694085, 23:08:05 DEBUG "624.5333333333333" 23:08:05 DEBUG ] 23:08:05 DEBUG } 23:08:05 DEBUG ] 23:08:05 DEBUG } 23:08:05 DEBUG } 23:08:05 INFO [http_cmd]: http status code OK 23:08:05 DEBUG --- status code --- 23:08:05 DEBUG http status code is 200 (expected 200) 23:08:05 DEBUG --- http response --- 23:08:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:05 DEBUG ----- output ----- 23:08:05 DEBUG { 23:08:05 DEBUG "status": "success", 23:08:05 DEBUG "data": { 23:08:05 DEBUG "resultType": "vector", 23:08:05 DEBUG "result": [ 23:08:05 DEBUG { 23:08:05 DEBUG "metric": { 23:08:05 DEBUG "pod": "ds-idrepo-2" 23:08:05 DEBUG }, 23:08:05 DEBUG "value": [ 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG [...] 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG "pod": "ds-idrepo-0" 23:08:05 DEBUG }, 23:08:05 DEBUG "value": [ 23:08:05 DEBUG 1686694085, 23:08:05 DEBUG "624.5333333333333" 23:08:05 DEBUG ] 23:08:05 DEBUG } 23:08:05 DEBUG ] 23:08:05 DEBUG } 23:08:05 DEBUG } 23:08:05 INFO [http_cmd]: http status code OK 23:08:05 INFO [http_cmd]: http status code OK 23:08:05 DEBUG --- status code --- 23:08:05 DEBUG --- status code --- 23:08:05 DEBUG http status code is 200 (expected 200) 23:08:05 DEBUG http status code is 200 (expected 200) 23:08:05 DEBUG --- http response --- 23:08:05 DEBUG --- http response --- 23:08:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:05 DEBUG ----- output ----- 23:08:05 DEBUG ----- output ----- 23:08:05 DEBUG { 23:08:05 DEBUG { 23:08:05 DEBUG "status": "success", 23:08:05 DEBUG "status": "success", 23:08:05 DEBUG "data": { 23:08:05 DEBUG "data": { 23:08:05 DEBUG "resultType": "vector", 23:08:05 DEBUG "resultType": "vector", 23:08:05 DEBUG "result": [ 23:08:05 DEBUG "result": [ 23:08:05 DEBUG { 23:08:05 DEBUG { 23:08:05 DEBUG "metric": { 23:08:05 DEBUG "metric": { 23:08:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:08:05 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:08:05 DEBUG }, 23:08:05 DEBUG }, 23:08:05 DEBUG "value": [ 23:08:05 DEBUG "value": [ 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG [...] 23:08:05 DEBUG [...] 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG 23:08:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:08:05 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:08:05 DEBUG }, 23:08:05 DEBUG }, 23:08:05 DEBUG "value": [ 23:08:05 DEBUG "value": [ 23:08:05 DEBUG 1686694085, 23:08:05 DEBUG 1686694085, 23:08:05 DEBUG "0" 23:08:05 DEBUG "0" 23:08:05 DEBUG ] 23:08:05 DEBUG ] 23:08:05 DEBUG } 23:08:05 DEBUG } 23:08:05 DEBUG ] 23:08:05 DEBUG ] 23:08:05 DEBUG } 23:08:05 DEBUG } 23:08:05 DEBUG } 23:08:05 DEBUG } 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694086" 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694086" 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694086" 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694086" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "0.2" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "0" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG ] 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG } 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG ] 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG } 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:06 DEBUG } 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "instance": "10.142.0.100:9100" 23:08:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:08:06 DEBUG }, 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "instance": "10.142.0.99:9100" 23:08:06 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 23:08:06 DEBUG }, 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "0" 23:08:06 DEBUG "0" 23:08:06 DEBUG ] 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694086" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "0" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694086" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "pod": "ds-cts-0" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "pod": "ds-idrepo-0" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "0" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694086" 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694086" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "pod": "ds-cts-0" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "pod": "ds-cts-2" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "0" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "0" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686694086" 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694086" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "13616865280" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "instance": "10.142.0.100:9100" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "instance": "10.142.0.99:9100" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "0.00030000000000048505" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694086" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "pod": "ds-idrepo-2" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "pod": "ds-idrepo-0" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "624.5" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694086" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "instance": "10.142.0.100:9100" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "instance": "10.142.0.99:9100" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "19336973.366666667" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694086" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "instance": "10.142.0.100:9100" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "instance": "10.142.0.99:9100" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "19336794.8" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694086" 23:08:06 INFO [http_cmd]: http status code OK 23:08:06 DEBUG --- status code --- 23:08:06 DEBUG http status code is 200 (expected 200) 23:08:06 DEBUG --- http response --- 23:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:06 DEBUG ----- output ----- 23:08:06 DEBUG { 23:08:06 DEBUG "status": "success", 23:08:06 DEBUG "data": { 23:08:06 DEBUG "resultType": "vector", 23:08:06 DEBUG "result": [ 23:08:06 DEBUG { 23:08:06 DEBUG "metric": { 23:08:06 DEBUG "pod": "ds-idrepo-2" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG [...] 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG 23:08:06 DEBUG "pod": "ds-idrepo-0" 23:08:06 DEBUG }, 23:08:06 DEBUG "value": [ 23:08:06 DEBUG 1686694086, 23:08:06 DEBUG "624.4666666666667" 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG ] 23:08:06 DEBUG } 23:08:06 DEBUG } 23:08:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694087" 23:08:07 INFO [http_cmd]: http status code OK 23:08:07 DEBUG --- status code --- 23:08:07 DEBUG http status code is 200 (expected 200) 23:08:07 DEBUG --- http response --- 23:08:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:08:07 DEBUG ----- output ----- 23:08:07 DEBUG { 23:08:07 DEBUG "status": "success", 23:08:07 DEBUG "data": { 23:08:07 DEBUG "resultType": "vector", 23:08:07 DEBUG "result": [ 23:08:07 DEBUG { 23:08:07 DEBUG "metric": { 23:08:07 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:08:07 DEBUG }, 23:08:07 DEBUG "value": [ 23:08:07 DEBUG 23:08:07 DEBUG 23:08:07 DEBUG 23:08:07 DEBUG [...] 23:08:07 DEBUG 23:08:07 DEBUG 23:08:07 DEBUG 23:08:07 DEBUG "pod": "login-ui-55f8bc5458-9zpvt" 23:08:07 DEBUG }, 23:08:07 DEBUG "value": [ 23:08:07 DEBUG 1686694087, 23:08:07 DEBUG "0" 23:08:07 DEBUG ] 23:08:07 DEBUG } 23:08:07 DEBUG ] 23:08:07 DEBUG } 23:08:07 DEBUG } 23:08:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686694087" 23:08:07 INFO [http_cmd]: http status code OK 23:08:07 DEBUG --- status code --- 23:08:07 DEBUG http status code is 200 (expected 200) 23:08:07 DEBUG --- http response --- 23:08:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:07 DEBUG ----- output ----- 23:08:07 DEBUG { 23:08:07 DEBUG "status": "success", 23:08:07 DEBUG "data": { 23:08:07 DEBUG "resultType": "vector", 23:08:07 DEBUG "result": [ 23:08:07 DEBUG { 23:08:07 DEBUG "metric": { 23:08:07 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:08:07 DEBUG }, 23:08:07 DEBUG "value": [ 23:08:07 DEBUG 23:08:07 DEBUG 23:08:07 DEBUG 23:08:07 DEBUG [...] 23:08:07 DEBUG 23:08:07 DEBUG 23:08:07 DEBUG 23:08:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:08:07 DEBUG }, 23:08:07 DEBUG "value": [ 23:08:07 DEBUG 1686694087, 23:08:07 DEBUG "0.23852057657052458" 23:08:07 DEBUG ] 23:08:07 DEBUG } 23:08:07 DEBUG ] 23:08:07 DEBUG } 23:08:07 DEBUG } 23:08:25 INFO 23:08:25 INFO [loop_until]: kubectl --namespace=xlou top pods 23:08:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:25 INFO [loop_until]: OK (rc = 0) 23:08:25 DEBUG --- stdout --- 23:08:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5760Mi am-7849cf7bdb-ntqzs 6m 5722Mi am-7849cf7bdb-xrngh 6m 5690Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 3064m 13806Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 15m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 2983m 5014Mi idm-58dc667486-rgz9j 2617m 5102Mi lodemon-6455d5ff7c-zzndq 2m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 200m 645Mi 23:08:25 DEBUG --- stderr --- 23:08:25 DEBUG 23:08:29 INFO 23:08:29 INFO [loop_until]: kubectl --namespace=xlou top node 23:08:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:29 INFO [loop_until]: OK (rc = 0) 23:08:29 DEBUG --- stdout --- 23:08:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 3041m 19% 5950Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 69m 0% 6860Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 74m 0% 6853Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 2655m 16% 6077Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 69m 0% 6949Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 439m 2% 3142Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 81m 0% 1483Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1167Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 63m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13732Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 3188m 20% 14535Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 61m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14614Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 266m 1% 1969Mi 3% 23:08:29 DEBUG --- stderr --- 23:08:29 DEBUG 23:09:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694145" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "ds-cts-2" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694145, 23:09:06 DEBUG "0" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694146" 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:09:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:09:06 DEBUG }, 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:09:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG }, 23:09:06 DEBUG }, 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG "value": [ 23:09:06 DEBUG "value": [ 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG "0" 23:09:06 DEBUG "0" 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:06 DEBUG ] 23:09:06 DEBUG ] 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 DEBUG { 23:09:06 DEBUG ] 23:09:06 DEBUG ] 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 DEBUG "data": { 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "ds-cts-0" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "ds-idrepo-0" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "585.3333333333334" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "ds-idrepo-2" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "ds-idrepo-0" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "585.3" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG } 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "instance": "10.142.0.100:9100" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "instance": "10.142.0.99:9100" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "0.0003333333333333262" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "ds-cts-2" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "ds-idrepo-2" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "0" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "0.2" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "0" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "ds-cts-2" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "0" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "ds-cts-0" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "ds-idrepo-0" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "0" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "ds-cts-0" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "ds-cts-2" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "0" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "0" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686694146" 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "13616283648" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "instance": "10.142.0.100:9100" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "instance": "10.142.0.99:9100" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "0.00046666666666510536" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "ds-idrepo-2" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "ds-idrepo-0" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "585.2666666666667" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "instance": "10.142.0.100:9100" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "instance": "10.142.0.99:9100" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "11335024.366666667" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "instance": "10.142.0.100:9100" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "instance": "10.142.0.99:9100" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "11330897.866666667" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694146" 23:09:06 INFO [http_cmd]: http status code OK 23:09:06 DEBUG --- status code --- 23:09:06 DEBUG http status code is 200 (expected 200) 23:09:06 DEBUG --- http response --- 23:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:06 DEBUG ----- output ----- 23:09:06 DEBUG { 23:09:06 DEBUG "status": "success", 23:09:06 DEBUG "data": { 23:09:06 DEBUG "resultType": "vector", 23:09:06 DEBUG "result": [ 23:09:06 DEBUG { 23:09:06 DEBUG "metric": { 23:09:06 DEBUG "pod": "ds-idrepo-2" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG [...] 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG 23:09:06 DEBUG "pod": "ds-idrepo-0" 23:09:06 DEBUG }, 23:09:06 DEBUG "value": [ 23:09:06 DEBUG 1686694146, 23:09:06 DEBUG "585.2666666666667" 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG ] 23:09:06 DEBUG } 23:09:06 DEBUG } 23:09:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694147" 23:09:07 INFO [http_cmd]: http status code OK 23:09:07 DEBUG --- status code --- 23:09:07 DEBUG http status code is 200 (expected 200) 23:09:07 DEBUG --- http response --- 23:09:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:09:07 DEBUG ----- output ----- 23:09:07 DEBUG { 23:09:07 DEBUG "status": "success", 23:09:07 DEBUG "data": { 23:09:07 DEBUG "resultType": "vector", 23:09:07 DEBUG "result": [ 23:09:07 DEBUG { 23:09:07 DEBUG "metric": { 23:09:07 DEBUG "pod": "ds-cts-2" 23:09:07 DEBUG }, 23:09:07 DEBUG "value": [ 23:09:07 DEBUG 23:09:07 DEBUG 23:09:07 DEBUG 23:09:07 DEBUG [...] 23:09:07 DEBUG 23:09:07 DEBUG 23:09:07 DEBUG 23:09:07 DEBUG "pod": "idm-58dc667486-ql6zm" 23:09:07 DEBUG }, 23:09:07 DEBUG "value": [ 23:09:07 DEBUG 1686694147, 23:09:07 DEBUG "0" 23:09:07 DEBUG ] 23:09:07 DEBUG } 23:09:07 DEBUG ] 23:09:07 DEBUG } 23:09:07 DEBUG } 23:09:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686694147" 23:09:07 INFO [http_cmd]: http status code OK 23:09:07 DEBUG --- status code --- 23:09:07 DEBUG http status code is 200 (expected 200) 23:09:07 DEBUG --- http response --- 23:09:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:07 DEBUG ----- output ----- 23:09:07 DEBUG { 23:09:07 DEBUG "status": "success", 23:09:07 DEBUG "data": { 23:09:07 DEBUG "resultType": "vector", 23:09:07 DEBUG "result": [ 23:09:07 DEBUG { 23:09:07 DEBUG "metric": { 23:09:07 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:09:07 DEBUG }, 23:09:07 DEBUG "value": [ 23:09:07 DEBUG 23:09:07 DEBUG 23:09:07 DEBUG 23:09:07 DEBUG [...] 23:09:07 DEBUG 23:09:07 DEBUG 23:09:07 DEBUG 23:09:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:09:07 DEBUG }, 23:09:07 DEBUG "value": [ 23:09:07 DEBUG 1686694147, 23:09:07 DEBUG "0.231656464977092" 23:09:07 DEBUG ] 23:09:07 DEBUG } 23:09:07 DEBUG ] 23:09:07 DEBUG } 23:09:07 DEBUG } 23:09:25 INFO 23:09:25 INFO [loop_until]: kubectl --namespace=xlou top pods 23:09:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:25 INFO [loop_until]: OK (rc = 0) 23:09:25 DEBUG --- stdout --- 23:09:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5761Mi am-7849cf7bdb-ntqzs 6m 5722Mi am-7849cf7bdb-xrngh 7m 5690Mi ds-cts-0 9m 417Mi ds-cts-1 8m 425Mi ds-cts-2 9m 393Mi ds-idrepo-0 1879m 13822Mi ds-idrepo-1 16m 13821Mi ds-idrepo-2 14m 12910Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 1427m 5014Mi idm-58dc667486-rgz9j 1534m 5101Mi lodemon-6455d5ff7c-zzndq 3m 71Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 181m 644Mi 23:09:25 DEBUG --- stderr --- 23:09:25 DEBUG 23:09:29 INFO 23:09:29 INFO [loop_until]: kubectl --namespace=xlou top node 23:09:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:29 INFO [loop_until]: OK (rc = 0) 23:09:29 DEBUG --- stdout --- 23:09:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 1047m 6% 5949Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 72m 0% 6838Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 978m 6% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6948Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 318m 2% 3125Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 84m 0% 1481Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 63m 0% 1170Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 62m 0% 1199Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13737Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 1866m 11% 14536Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 62m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 77m 0% 14629Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 193m 1% 1967Mi 3% 23:09:29 DEBUG --- stderr --- 23:09:29 DEBUG 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "pod": "ds-cts-2" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "0.002870586112959749" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG { 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:06 DEBUG { 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "data": { 23:10:06 DEBUG { 23:10:06 DEBUG "data": { 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG "result": [ 23:10:06 DEBUG "data": { 23:10:06 DEBUG "result": [ 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG { 23:10:06 DEBUG { 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "result": [ 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:10:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:10:06 DEBUG { 23:10:06 DEBUG "pod": "ds-cts-0" 23:10:06 DEBUG "pod": "ds-idrepo-2" 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG }, 23:10:06 DEBUG }, 23:10:06 DEBUG "metric": { 23:10:06 DEBUG }, 23:10:06 DEBUG }, 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "value": [ 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG }, 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG { 23:10:06 DEBUG { 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG [...] 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG [...] 23:10:06 DEBUG "data": { 23:10:06 DEBUG "data": { 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "result": [ 23:10:06 DEBUG "result": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG { 23:10:06 DEBUG { 23:10:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:10:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:10:06 DEBUG 23:10:06 DEBUG "pod": "ds-idrepo-0" 23:10:06 DEBUG "pod": "ds-idrepo-0" 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG }, 23:10:06 DEBUG }, 23:10:06 DEBUG 23:10:06 DEBUG }, 23:10:06 DEBUG }, 23:10:06 DEBUG "instance": "10.142.0.100:9100" 23:10:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "value": [ 23:10:06 DEBUG }, 23:10:06 DEBUG }, 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG }, 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "0" 23:10:06 DEBUG "0.2" 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "0" 23:10:06 DEBUG "0" 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG ] 23:10:06 DEBUG ] 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG ] 23:10:06 DEBUG ] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG "0" 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG ] 23:10:06 DEBUG ] 23:10:06 DEBUG ] 23:10:06 DEBUG ] 23:10:06 DEBUG ] 23:10:06 DEBUG [...] 23:10:06 DEBUG [...] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG } 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG } 23:10:06 DEBUG "instance": "10.142.0.99:9100" 23:10:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:10:06 DEBUG }, 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "0" 23:10:06 DEBUG "0" 23:10:06 DEBUG ] 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "pod": "ds-cts-2" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "0" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "pod": "ds-cts-0" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "pod": "ds-idrepo-0" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "0" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "pod": "ds-cts-2" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "157061101.4547956" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "0" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "pod": "ds-cts-0" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "pod": "ds-cts-2" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "0" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "instance": "10.142.0.100:9100" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "instance": "10.142.0.99:9100" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "0.00023333333333350008" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "13619445760" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "pod": "ds-idrepo-2" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "pod": "ds-idrepo-0" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "0" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "instance": "10.142.0.100:9100" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "instance": "10.142.0.99:9100" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "11062" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "instance": "10.142.0.100:9100" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "instance": "10.142.0.99:9100" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "6872.266666666666" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694206" 23:10:06 INFO [http_cmd]: http status code OK 23:10:06 DEBUG --- status code --- 23:10:06 DEBUG http status code is 200 (expected 200) 23:10:06 DEBUG --- http response --- 23:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:06 DEBUG ----- output ----- 23:10:06 DEBUG { 23:10:06 DEBUG "status": "success", 23:10:06 DEBUG "data": { 23:10:06 DEBUG "resultType": "vector", 23:10:06 DEBUG "result": [ 23:10:06 DEBUG { 23:10:06 DEBUG "metric": { 23:10:06 DEBUG "pod": "ds-idrepo-2" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG [...] 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG 23:10:06 DEBUG "pod": "ds-idrepo-0" 23:10:06 DEBUG }, 23:10:06 DEBUG "value": [ 23:10:06 DEBUG 1686694206, 23:10:06 DEBUG "0" 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG ] 23:10:06 DEBUG } 23:10:06 DEBUG } 23:10:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694207" 23:10:07 INFO [http_cmd]: http status code OK 23:10:07 DEBUG --- status code --- 23:10:07 DEBUG http status code is 200 (expected 200) 23:10:07 DEBUG --- http response --- 23:10:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:10:07 DEBUG ----- output ----- 23:10:07 DEBUG { 23:10:07 DEBUG "status": "success", 23:10:07 DEBUG "data": { 23:10:07 DEBUG "resultType": "vector", 23:10:07 DEBUG "result": [ 23:10:07 DEBUG { 23:10:07 DEBUG "metric": { 23:10:07 DEBUG "pod": "ds-cts-2" 23:10:07 DEBUG }, 23:10:07 DEBUG "value": [ 23:10:07 DEBUG 23:10:07 DEBUG 23:10:07 DEBUG 23:10:07 DEBUG [...] 23:10:07 DEBUG 23:10:07 DEBUG 23:10:07 DEBUG 23:10:07 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:10:07 DEBUG }, 23:10:07 DEBUG "value": [ 23:10:07 DEBUG 1686694207, 23:10:07 DEBUG "0" 23:10:07 DEBUG ] 23:10:07 DEBUG } 23:10:07 DEBUG ] 23:10:07 DEBUG } 23:10:07 DEBUG } 23:10:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686694207" 23:10:07 INFO [http_cmd]: http status code OK 23:10:07 DEBUG --- status code --- 23:10:07 DEBUG http status code is 200 (expected 200) 23:10:07 DEBUG --- http response --- 23:10:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:07 DEBUG ----- output ----- 23:10:07 DEBUG { 23:10:07 DEBUG "status": "success", 23:10:07 DEBUG "data": { 23:10:07 DEBUG "resultType": "vector", 23:10:07 DEBUG "result": [ 23:10:07 DEBUG { 23:10:07 DEBUG "metric": { 23:10:07 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:10:07 DEBUG }, 23:10:07 DEBUG "value": [ 23:10:07 DEBUG 23:10:07 DEBUG 23:10:07 DEBUG 23:10:07 DEBUG [...] 23:10:07 DEBUG 23:10:07 DEBUG 23:10:07 DEBUG 23:10:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:10:07 DEBUG }, 23:10:07 DEBUG "value": [ 23:10:07 DEBUG 1686694207, 23:10:07 DEBUG "0.08807453294139363" 23:10:07 DEBUG ] 23:10:07 DEBUG } 23:10:07 DEBUG ] 23:10:07 DEBUG } 23:10:07 DEBUG } 23:10:25 INFO 23:10:25 INFO [loop_until]: kubectl --namespace=xlou top pods 23:10:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:25 INFO [loop_until]: OK (rc = 0) 23:10:25 DEBUG --- stdout --- 23:10:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 4Mi am-7849cf7bdb-lzlql 8m 5761Mi am-7849cf7bdb-ntqzs 6m 5722Mi am-7849cf7bdb-xrngh 9m 5690Mi ds-cts-0 12m 418Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 17m 13797Mi ds-idrepo-1 17m 13820Mi ds-idrepo-2 15m 12906Mi end-user-ui-5c589c665b-7tbkq 1m 3Mi idm-58dc667486-ql6zm 5m 5013Mi idm-58dc667486-rgz9j 6m 5100Mi lodemon-6455d5ff7c-zzndq 5m 70Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 2m 131Mi 23:10:25 DEBUG --- stderr --- 23:10:25 DEBUG 23:10:29 INFO 23:10:29 INFO [loop_until]: kubectl --namespace=xlou top node 23:10:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:29 INFO [loop_until]: OK (rc = 0) 23:10:29 DEBUG --- stdout --- 23:10:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 67m 0% 5952Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6862Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 68m 0% 6841Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 71m 0% 6081Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 70m 0% 6951Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 225m 1% 3136Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 83m 0% 1484Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 60m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 65m 0% 1198Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 72m 0% 13728Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 72m 0% 14511Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 65m 0% 1165Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 73m 0% 14618Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 74m 0% 1461Mi 2% 23:10:29 DEBUG --- stderr --- 23:10:29 DEBUG 127.0.0.1 - - [13/Jun/2023 23:11:00] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "ds-cts-2" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0.00041216544608667393" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "ds-cts-2" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0.03333333333333333" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "ds-cts-0" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "ds-idrepo-0" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0.2" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0.03333333333333333" 23:11:06 DEBUG ] 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG } 23:11:06 DEBUG --- status code --- 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG ] 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG } 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG } 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:06 DEBUG { 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG { 23:11:06 DEBUG "data": { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "result": [ 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG { 23:11:06 DEBUG "result": [ 23:11:06 DEBUG "metric": { 23:11:06 DEBUG { 23:11:06 DEBUG "instance": "10.142.0.100:9100" 23:11:06 DEBUG "metric": { 23:11:06 DEBUG }, 23:11:06 DEBUG "pod": "ds-idrepo-2" 23:11:06 DEBUG "value": [ 23:11:06 DEBUG }, 23:11:06 DEBUG 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "instance": "10.142.0.99:9100" 23:11:06 DEBUG 23:11:06 DEBUG }, 23:11:06 DEBUG "pod": "ds-idrepo-0" 23:11:06 DEBUG "value": [ 23:11:06 DEBUG }, 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG "0" 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG ] 23:11:06 DEBUG "0" 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "ds-cts-0" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "ds-idrepo-0" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "ds-cts-2" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "221.04812932319336" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0.05908111111111111" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "ds-cts-0" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "ds-cts-2" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "instance": "10.142.0.100:9100" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "instance": "10.142.0.99:9100" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0.0003333333333330302" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "13610823680" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "ds-idrepo-2" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "ds-idrepo-0" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694266" 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694266" 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 INFO [http_cmd]: http status code OK 127.0.0.1 - - [13/Jun/2023 23:11:06] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 23:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694266" 23:11:06 DEBUG { 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG "data": { 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:06 DEBUG "result": [ 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "instance": "10.142.0.100:9100" 23:11:06 DEBUG "data": { 23:11:06 DEBUG }, 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "value": [ 23:11:06 DEBUG "result": [ 23:11:06 DEBUG 23:11:06 DEBUG { 23:11:06 DEBUG 23:11:06 DEBUG "metric": { 23:11:06 DEBUG 23:11:06 DEBUG "instance": "10.142.0.100:9100" 23:11:06 DEBUG [...] 23:11:06 DEBUG }, 23:11:06 DEBUG 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "instance": "10.142.0.99:9100" 23:11:06 DEBUG 23:11:06 DEBUG }, 23:11:06 DEBUG [...] 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG 23:11:06 DEBUG "10623.533333333333" 23:11:06 DEBUG 23:11:06 DEBUG ] 23:11:06 DEBUG "instance": "10.142.0.99:9100" 23:11:06 DEBUG } 23:11:06 DEBUG }, 23:11:06 DEBUG ] 23:11:06 DEBUG "value": [ 23:11:06 DEBUG } 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG } 23:11:06 DEBUG "6474.733333333334" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:06 INFO [http_cmd]: http status code OK 23:11:06 DEBUG --- status code --- 23:11:06 DEBUG http status code is 200 (expected 200) 23:11:06 DEBUG --- http response --- 23:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:06 DEBUG ----- output ----- 23:11:06 DEBUG { 23:11:06 DEBUG "status": "success", 23:11:06 DEBUG "data": { 23:11:06 DEBUG "resultType": "vector", 23:11:06 DEBUG "result": [ 23:11:06 DEBUG { 23:11:06 DEBUG "metric": { 23:11:06 DEBUG "pod": "ds-idrepo-2" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG [...] 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG 23:11:06 DEBUG "pod": "ds-idrepo-0" 23:11:06 DEBUG }, 23:11:06 DEBUG "value": [ 23:11:06 DEBUG 1686694266, 23:11:06 DEBUG "0" 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG ] 23:11:06 DEBUG } 23:11:06 DEBUG } 23:11:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694267" 23:11:07 INFO [http_cmd]: http status code OK 23:11:07 DEBUG --- status code --- 23:11:07 DEBUG http status code is 200 (expected 200) 23:11:07 DEBUG --- http response --- 23:11:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:11:07 DEBUG ----- output ----- 23:11:07 DEBUG { 23:11:07 DEBUG "status": "success", 23:11:07 DEBUG "data": { 23:11:07 DEBUG "resultType": "vector", 23:11:07 DEBUG "result": [ 23:11:07 DEBUG { 23:11:07 DEBUG "metric": { 23:11:07 DEBUG "pod": "ds-cts-2" 23:11:07 DEBUG }, 23:11:07 DEBUG "value": [ 23:11:07 DEBUG 23:11:07 DEBUG 23:11:07 DEBUG 23:11:07 DEBUG [...] 23:11:07 DEBUG 23:11:07 DEBUG 23:11:07 DEBUG 23:11:07 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:11:07 DEBUG }, 23:11:07 DEBUG "value": [ 23:11:07 DEBUG 1686694267, 23:11:07 DEBUG "0" 23:11:07 DEBUG ] 23:11:07 DEBUG } 23:11:07 DEBUG ] 23:11:07 DEBUG } 23:11:07 DEBUG } 23:11:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686694267" 23:11:07 INFO [http_cmd]: http status code OK 23:11:07 DEBUG --- status code --- 23:11:07 DEBUG http status code is 200 (expected 200) 23:11:07 DEBUG --- http response --- 23:11:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:07 DEBUG ----- output ----- 23:11:07 DEBUG { 23:11:07 DEBUG "status": "success", 23:11:07 DEBUG "data": { 23:11:07 DEBUG "resultType": "vector", 23:11:07 DEBUG "result": [ 23:11:07 DEBUG { 23:11:07 DEBUG "metric": { 23:11:07 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:11:07 DEBUG }, 23:11:07 DEBUG "value": [ 23:11:07 DEBUG 23:11:07 DEBUG 23:11:07 DEBUG 23:11:07 DEBUG [...] 23:11:07 DEBUG 23:11:07 DEBUG 23:11:07 DEBUG 23:11:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:11:07 DEBUG }, 23:11:07 DEBUG "value": [ 23:11:07 DEBUG 1686694267, 23:11:07 DEBUG "0.026000839346872383" 23:11:07 DEBUG ] 23:11:07 DEBUG } 23:11:07 DEBUG ] 23:11:07 DEBUG } 23:11:07 DEBUG } 127.0.0.1 - - [13/Jun/2023 23:11:13] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:11:19] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:11:25] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 23:11:25 INFO 23:11:25 INFO [loop_until]: kubectl --namespace=xlou top pods 23:11:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:25 INFO [loop_until]: OK (rc = 0) 23:11:25 DEBUG --- stdout --- 23:11:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 5Mi am-7849cf7bdb-lzlql 8m 5767Mi am-7849cf7bdb-ntqzs 6m 5722Mi am-7849cf7bdb-xrngh 7m 5690Mi ds-cts-0 9m 417Mi ds-cts-1 68m 425Mi ds-cts-2 68m 393Mi ds-idrepo-0 513m 13770Mi ds-idrepo-1 210m 13753Mi ds-idrepo-2 333m 12905Mi end-user-ui-5c589c665b-7tbkq 1m 5Mi idm-58dc667486-ql6zm 4m 5013Mi idm-58dc667486-rgz9j 5m 5100Mi lodemon-6455d5ff7c-zzndq 12m 73Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 874m 523Mi 23:11:25 DEBUG --- stderr --- 23:11:25 DEBUG 23:11:29 INFO 23:11:29 INFO [loop_until]: kubectl --namespace=xlou top node 23:11:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:30 INFO [loop_until]: OK (rc = 0) 23:11:30 DEBUG --- stdout --- 23:11:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 70m 0% 5953Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 71m 0% 6869Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 69m 0% 6838Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 72m 0% 6078Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 77m 0% 6953Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 229m 1% 3145Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 93m 0% 1489Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1169Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 72m 0% 1202Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 362m 2% 13800Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 521m 3% 14489Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 67m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 332m 2% 14565Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 649m 4% 1853Mi 3% 23:11:30 DEBUG --- stderr --- 23:11:30 DEBUG 127.0.0.1 - - [13/Jun/2023 23:11:31] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:11:37] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:11:44] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:11:50] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:11:56] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:12:02] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:12:06 DEBUG "pod": "ds-cts-2" 23:12:06 DEBUG }, 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:12:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:12:06 DEBUG }, 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG "0.000034819372514956205" 23:12:06 DEBUG ] 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694326" 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "ds-cts-2" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "instance": "10.142.0.100:9100" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "instance": "10.142.0.99:9100" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "ds-cts-0" 23:12:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:12:06 DEBUG }, 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "ds-idrepo-0" 23:12:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:12:06 DEBUG }, 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG "0.2" 23:12:06 DEBUG ] 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "ds-idrepo-2" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "ds-idrepo-0" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "ds-cts-2" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "ds-cts-0" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "ds-idrepo-0" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "am-7849cf7bdb-ntqzs" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "am-7849cf7bdb-xrngh" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "ds-cts-0" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "ds-cts-2" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694326" 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "instance": "10.142.0.100:9100" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "instance": "10.142.0.99:9100" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0.0001999999999990602" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-5msf" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "13611327488" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "ds-idrepo-2" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "ds-idrepo-0" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694326" 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "instance": "10.142.0.100:9100" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "instance": "10.142.0.99:9100" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "8681.5" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "pod": "ds-idrepo-2" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "pod": "ds-idrepo-0" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "0" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1686694326" 23:12:06 INFO [http_cmd]: http status code OK 23:12:06 DEBUG --- status code --- 23:12:06 DEBUG http status code is 200 (expected 200) 23:12:06 DEBUG --- http response --- 23:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:06 DEBUG ----- output ----- 23:12:06 DEBUG { 23:12:06 DEBUG "status": "success", 23:12:06 DEBUG "data": { 23:12:06 DEBUG "resultType": "vector", 23:12:06 DEBUG "result": [ 23:12:06 DEBUG { 23:12:06 DEBUG "metric": { 23:12:06 DEBUG "instance": "10.142.0.100:9100" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG [...] 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG 23:12:06 DEBUG "instance": "10.142.0.99:9100" 23:12:06 DEBUG }, 23:12:06 DEBUG "value": [ 23:12:06 DEBUG 1686694326, 23:12:06 DEBUG "10733.333333333332" 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG ] 23:12:06 DEBUG } 23:12:06 DEBUG } 23:12:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1686694327" 23:12:07 INFO [http_cmd]: http status code OK 23:12:07 DEBUG --- status code --- 23:12:07 DEBUG http status code is 200 (expected 200) 23:12:07 DEBUG --- http response --- 23:12:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:12:07 DEBUG ----- output ----- 23:12:07 DEBUG { 23:12:07 DEBUG "status": "success", 23:12:07 DEBUG "data": { 23:12:07 DEBUG "resultType": "vector", 23:12:07 DEBUG "result": [ 23:12:07 DEBUG { 23:12:07 DEBUG "metric": { 23:12:07 DEBUG "pod": "ds-cts-2" 23:12:07 DEBUG }, 23:12:07 DEBUG "value": [ 23:12:07 DEBUG 23:12:07 DEBUG 23:12:07 DEBUG 23:12:07 DEBUG [...] 23:12:07 DEBUG 23:12:07 DEBUG 23:12:07 DEBUG 23:12:07 DEBUG "pod": "admin-ui-7657fd47d8-q5vwf" 23:12:07 DEBUG }, 23:12:07 DEBUG "value": [ 23:12:07 DEBUG 1686694327, 23:12:07 DEBUG "0" 23:12:07 DEBUG ] 23:12:07 DEBUG } 23:12:07 DEBUG ] 23:12:07 DEBUG } 23:12:07 DEBUG } 23:12:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1686694327" 23:12:07 INFO [http_cmd]: http status code OK 23:12:07 DEBUG --- status code --- 23:12:07 DEBUG http status code is 200 (expected 200) 23:12:07 DEBUG --- http response --- 23:12:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:07 DEBUG ----- output ----- 23:12:07 DEBUG { 23:12:07 DEBUG "status": "success", 23:12:07 DEBUG "data": { 23:12:07 DEBUG "resultType": "vector", 23:12:07 DEBUG "result": [ 23:12:07 DEBUG { 23:12:07 DEBUG "metric": { 23:12:07 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-s6rh" 23:12:07 DEBUG }, 23:12:07 DEBUG "value": [ 23:12:07 DEBUG 23:12:07 DEBUG 23:12:07 DEBUG 23:12:07 DEBUG [...] 23:12:07 DEBUG 23:12:07 DEBUG 23:12:07 DEBUG 23:12:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-h3n6" 23:12:07 DEBUG }, 23:12:07 DEBUG "value": [ 23:12:07 DEBUG 1686694327, 23:12:07 DEBUG "1.2306921223452048" 23:12:07 DEBUG ] 23:12:07 DEBUG } 23:12:07 DEBUG ] 23:12:07 DEBUG } 23:12:07 DEBUG } 127.0.0.1 - - [13/Jun/2023 23:12:09] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:12:15] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:12:21] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 23:12:25 INFO 23:12:25 INFO [loop_until]: kubectl --namespace=xlou top pods 23:12:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:26 INFO [loop_until]: OK (rc = 0) 23:12:26 DEBUG --- stdout --- 23:12:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7657fd47d8-q5vwf 1m 5Mi am-7849cf7bdb-lzlql 9m 5767Mi am-7849cf7bdb-ntqzs 6m 5722Mi am-7849cf7bdb-xrngh 6m 5690Mi ds-cts-0 9m 417Mi ds-cts-1 9m 425Mi ds-cts-2 8m 393Mi ds-idrepo-0 16m 13770Mi ds-idrepo-1 16m 13752Mi ds-idrepo-2 15m 12906Mi end-user-ui-5c589c665b-7tbkq 1m 5Mi idm-58dc667486-ql6zm 4m 5013Mi idm-58dc667486-rgz9j 6m 5100Mi lodemon-6455d5ff7c-zzndq 21m 74Mi login-ui-55f8bc5458-9zpvt 1m 3Mi overseer-0-7bb59f88f9-gcrrq 1000m 709Mi 23:12:26 DEBUG --- stderr --- 23:12:26 DEBUG 127.0.0.1 - - [13/Jun/2023 23:12:27] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 23:12:30 INFO 23:12:30 INFO [loop_until]: kubectl --namespace=xlou top node 23:12:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:30 INFO [loop_until]: OK (rc = 0) 23:12:30 DEBUG --- stdout --- 23:12:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-5mhf 70m 0% 5951Mi 10% gke-xlou-cdm-default-pool-f05840a3-6hll 68m 0% 6869Mi 11% gke-xlou-cdm-default-pool-f05840a3-83ll 70m 0% 6840Mi 11% gke-xlou-cdm-default-pool-f05840a3-8bck 69m 0% 6081Mi 10% gke-xlou-cdm-default-pool-f05840a3-b4tr 71m 0% 6947Mi 11% gke-xlou-cdm-default-pool-f05840a3-m0lk 223m 1% 3153Mi 5% gke-xlou-cdm-default-pool-f05840a3-s6rh 101m 0% 1488Mi 2% gke-xlou-cdm-ds-32e4dcb1-015h 61m 0% 1166Mi 1% gke-xlou-cdm-ds-32e4dcb1-0hks 64m 0% 1200Mi 2% gke-xlou-cdm-ds-32e4dcb1-5msf 70m 0% 13731Mi 23% gke-xlou-cdm-ds-32e4dcb1-jr09 71m 0% 14492Mi 24% gke-xlou-cdm-ds-32e4dcb1-vg18 63m 0% 1168Mi 1% gke-xlou-cdm-ds-32e4dcb1-vhcr 72m 0% 14556Mi 24% gke-xlou-cdm-frontend-a8771548-h3n6 1075m 6% 2052Mi 3% 23:12:30 DEBUG --- stderr --- 23:12:30 DEBUG 127.0.0.1 - - [13/Jun/2023 23:12:34] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:12:40] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 23:12:42 INFO Finished: True 23:12:42 INFO Waiting for threads to register finish flag 23:13:30 INFO Done. Have a nice day! :) 127.0.0.1 - - [13/Jun/2023 23:13:30] "GET /monitoring/stop HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:13:30] "GET /monitoring/average?start_time=23-06-13_15:58:06&stop_time=23-06-13_16:06:34 HTTP/1.1" 200 - 23:13:37 INFO File /tmp/lodemon_data-23-06-13_16:32:10/am_cts_reaper_cache_size.json does not exist. Skipping... 23:13:37 INFO File /tmp/lodemon_data-23-06-13_16:32:10/am_cts_reaper_search_seconds_total.json does not exist. Skipping... 23:13:37 INFO File /tmp/lodemon_data-23-06-13_16:32:10/ds_replication_replica_replayed_updates_conflicts_resolved.json does not exist. Skipping... 23:13:37 INFO File /tmp/lodemon_data-23-06-13_16:32:10/node_disk_read_bytes_total.json does not exist. Skipping... 23:13:37 INFO File /tmp/lodemon_data-23-06-13_16:32:10/node_disk_written_bytes_total.json does not exist. Skipping... 23:13:37 INFO File /tmp/lodemon_data-23-06-13_16:32:10/ds_backend_entry_count.json does not exist. Skipping... 127.0.0.1 - - [13/Jun/2023 23:13:45] "GET /monitoring/process HTTP/1.1" 200 - 127.0.0.1 - - [13/Jun/2023 23:13:45] "GET /monitoring/average?start_time=23-06-13_16:10:34&stop_time=23-06-13_22:08:00 HTTP/1.1" 200 -