==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-765d5f4748-dps7q Namespace: xlou Priority: 0 Node: gke-xlou-cdm-default-pool-cab466d1-h89r/10.142.0.26 Start Time: Thu, 18 May 2023 22:31:39 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=765d5f4748 skaffold.dev/run-id=652da061-3079-4c4f-aa10-ef00722b2f37 Annotations: Status: Running IP: 10.106.40.196 IPs: IP: 10.106.40.196 Controlled By: ReplicaSet/lodemon-765d5f4748 Containers: lodemon: Container ID: containerd://840452b752d8c565c0378441220adf28c78cabca3e8ab349850506cb04301767 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:21aa7819571d93a7f3c2e1355b96b08ae4717ae2 Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:b4cd4ad79a4fe740bf283ecfb348b507abe0e3672cdd540fd5aedaa47acc0a7b Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Thu, 18 May 2023 22:31:40 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9rlcm (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-9rlcm: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 04:27:49 INFO 04:27:49 INFO [loop_until]: kubectl --namespace=xlou top pods 04:27:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:27:49 INFO [loop_until]: OK (rc = 0) 04:27:49 DEBUG --- stdout --- 04:27:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 6793m 9180Mi am-7849cf7bdb-jcbb8 6353m 9165Mi am-7849cf7bdb-zrhp7 7115m 9197Mi ds-cts-0 8792m 13879Mi ds-cts-1 9304m 13827Mi ds-cts-2 9103m 13850Mi ds-idrepo-0 4489m 13823Mi ds-idrepo-1 13m 13825Mi ds-idrepo-2 16m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 8m 3245Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1923m 593Mi 04:27:49 DEBUG --- stderr --- 04:27:49 DEBUG 04:27:53 INFO 04:27:53 INFO [loop_until]: kubectl --namespace=xlou top node 04:27:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:27:53 INFO [loop_until]: OK (rc = 0) 04:27:53 DEBUG --- stdout --- 04:27:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7282m 45% 11358Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 82m 0% 4888Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 7775m 48% 11424Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 12831m 80% 11825Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 182m 1% 3020Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 86m 0% 1328Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5166Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 65m 0% 14803Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 9260m 58% 14654Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 65m 0% 14868Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4702m 29% 14803Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 8988m 56% 14648Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 9038m 56% 14689Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2046m 12% 2596Mi 4% 04:27:53 DEBUG --- stderr --- 04:27:53 DEBUG 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "ds-cts-2" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "ds-idrepo-2" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "ds-idrepo-0" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "ds-idrepo-2" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "ds-idrepo-0" 04:28:36 DEBUG "pod": "ds-cts-2" 04:28:36 DEBUG }, 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "ds-idrepo-2" 04:28:36 DEBUG "pod": "ds-idrepo-2" 04:28:36 DEBUG }, 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0" 04:28:36 DEBUG "0" 04:28:36 DEBUG ] 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "685.3" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:28:36 DEBUG "instance": "10.142.0.17:9100" 04:28:36 DEBUG }, 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684466916" 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:28:36 DEBUG "instance": "10.142.0.61:9100" 04:28:36 DEBUG }, 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "685.2" 04:28:36 DEBUG "0.0017999999999991209" 04:28:36 DEBUG ] 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684466916" 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "342.56666666666666" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "instance": "10.142.0.17:9100" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "instance": "10.142.0.61:9100" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "7419077.033333333" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "data": { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "pod": "ds-cts-2" 04:28:36 DEBUG }, 04:28:36 DEBUG "result": [ 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG { 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG "metric": { 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG }, 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG "value": [ 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:28:36 DEBUG [...] 04:28:36 DEBUG "pod": "ds-idrepo-2" 04:28:36 DEBUG }, 04:28:36 DEBUG 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0.2" 04:28:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:28:36 DEBUG "0" 04:28:36 DEBUG ] 04:28:36 DEBUG }, 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG "value": [ 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG "14602543104" 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684466916" 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG { 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:36 DEBUG "data": { 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG { 04:28:36 DEBUG "result": [ 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG { 04:28:36 DEBUG "data": { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "instance": "10.142.0.17:9100" 04:28:36 DEBUG "result": [ 04:28:36 DEBUG }, 04:28:36 DEBUG { 04:28:36 DEBUG "value": [ 04:28:36 DEBUG "metric": { 04:28:36 DEBUG 04:28:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:28:36 DEBUG 04:28:36 DEBUG }, 04:28:36 DEBUG 04:28:36 DEBUG "value": [ 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG "instance": "10.142.0.61:9100" 04:28:36 DEBUG 04:28:36 DEBUG }, 04:28:36 DEBUG 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:28:36 DEBUG "7421594.9" 04:28:36 DEBUG }, 04:28:36 DEBUG ] 04:28:36 DEBUG "value": [ 04:28:36 DEBUG } 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG ] 04:28:36 DEBUG "1.9882044903468639" 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "342.8666666666667" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0.30381713120753284" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "pod": "ds-idrepo-0" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "pod": "ds-idrepo-2" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684466916" 04:28:36 INFO [http_cmd]: http status code OK 04:28:36 DEBUG --- status code --- 04:28:36 DEBUG http status code is 200 (expected 200) 04:28:36 DEBUG --- http response --- 04:28:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:36 DEBUG ----- output ----- 04:28:36 DEBUG { 04:28:36 DEBUG "status": "success", 04:28:36 DEBUG "data": { 04:28:36 DEBUG "resultType": "vector", 04:28:36 DEBUG "result": [ 04:28:36 DEBUG { 04:28:36 DEBUG "metric": { 04:28:36 DEBUG "instance": "10.142.0.17:9100" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG [...] 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG 04:28:36 DEBUG "instance": "10.142.0.61:9100" 04:28:36 DEBUG }, 04:28:36 DEBUG "value": [ 04:28:36 DEBUG 1684466916, 04:28:36 DEBUG "0.000999999999999742" 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG ] 04:28:36 DEBUG } 04:28:36 DEBUG } 04:28:49 INFO 04:28:49 INFO [loop_until]: kubectl --namespace=xlou top pods 04:28:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:28:49 INFO [loop_until]: OK (rc = 0) 04:28:49 DEBUG --- stdout --- 04:28:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 5140m 9180Mi am-7849cf7bdb-jcbb8 5010m 9165Mi am-7849cf7bdb-zrhp7 4830m 9197Mi ds-cts-0 6841m 13836Mi ds-cts-1 6393m 13816Mi ds-cts-2 5763m 13847Mi ds-idrepo-0 3715m 13823Mi ds-idrepo-1 14m 13826Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 5m 3272Mi idm-78f4b47cb9-vh9b7 8m 3251Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1333m 95Mi 04:28:49 DEBUG --- stderr --- 04:28:49 DEBUG 04:28:53 INFO 04:28:53 INFO [loop_until]: kubectl --namespace=xlou top node 04:28:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:28:53 INFO [loop_until]: OK (rc = 0) 04:28:53 DEBUG --- stdout --- 04:28:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 3065m 19% 11356Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 81m 0% 4890Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 5113m 32% 11426Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7896m 49% 11789Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 157m 0% 3034Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 86m 0% 1331Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 92m 0% 5164Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 69m 0% 14804Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 4854m 30% 14628Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 68m 0% 14870Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 2460m 15% 14800Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 6172m 38% 14652Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 5888m 37% 14650Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 1183m 7% 2096Mi 3% 04:28:53 DEBUG --- stderr --- 04:28:53 DEBUG 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "ds-cts-2" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "ds-idrepo-2" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "ds-idrepo-0" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "ds-idrepo-2" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "ds-idrepo-0" 04:29:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:29:36 DEBUG }, 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "ds-idrepo-2" 04:29:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:29:36 DEBUG }, 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "ds-cts-2" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "ds-idrepo-2" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "instance": "10.142.0.17:9100" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "instance": "10.142.0.61:9100" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0.0020666666666670605" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "ds-cts-2" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "instance": "10.142.0.17:9100" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "instance": "10.142.0.61:9100" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "2425445.8666666667" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684466976" 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG --- http response --- 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG { 04:29:36 DEBUG { 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:36 DEBUG "data": { 04:29:36 DEBUG "data": { 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG { 04:29:36 DEBUG "result": [ 04:29:36 DEBUG "result": [ 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG { 04:29:36 DEBUG { 04:29:36 DEBUG "data": { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:29:36 DEBUG "pod": "ds-cts-2" 04:29:36 DEBUG "result": [ 04:29:36 DEBUG }, 04:29:36 DEBUG }, 04:29:36 DEBUG { 04:29:36 DEBUG "value": [ 04:29:36 DEBUG "value": [ 04:29:36 DEBUG "metric": { 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG }, 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "value": [ 04:29:36 DEBUG [...] 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:29:36 DEBUG "pod": "ds-idrepo-2" 04:29:36 DEBUG 04:29:36 DEBUG }, 04:29:36 DEBUG }, 04:29:36 DEBUG 04:29:36 DEBUG "value": [ 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:29:36 DEBUG "0.2" 04:29:36 DEBUG "0" 04:29:36 DEBUG }, 04:29:36 DEBUG ] 04:29:36 DEBUG ] 04:29:36 DEBUG "value": [ 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG ] 04:29:36 DEBUG ] 04:29:36 DEBUG "14597357568" 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "ds-cts-2" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684466976" 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "instance": "10.142.0.17:9100" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "instance": "10.142.0.61:9100" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "2426124.9" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0.029204725080769028" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0.5693312680186497" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "pod": "ds-idrepo-0" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "pod": "ds-idrepo-2" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684466976" 04:29:36 INFO [http_cmd]: http status code OK 04:29:36 DEBUG --- status code --- 04:29:36 DEBUG http status code is 200 (expected 200) 04:29:36 DEBUG --- http response --- 04:29:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:36 DEBUG ----- output ----- 04:29:36 DEBUG { 04:29:36 DEBUG "status": "success", 04:29:36 DEBUG "data": { 04:29:36 DEBUG "resultType": "vector", 04:29:36 DEBUG "result": [ 04:29:36 DEBUG { 04:29:36 DEBUG "metric": { 04:29:36 DEBUG "instance": "10.142.0.17:9100" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG [...] 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG 04:29:36 DEBUG "instance": "10.142.0.61:9100" 04:29:36 DEBUG }, 04:29:36 DEBUG "value": [ 04:29:36 DEBUG 1684466976, 04:29:36 DEBUG "0.0003333333333333854" 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG ] 04:29:36 DEBUG } 04:29:36 DEBUG } 04:29:49 INFO 04:29:49 INFO [loop_until]: kubectl --namespace=xlou top pods 04:29:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:29:49 INFO [loop_until]: OK (rc = 0) 04:29:49 DEBUG --- stdout --- 04:29:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 2085m 9183Mi am-7849cf7bdb-jcbb8 2454m 9166Mi am-7849cf7bdb-zrhp7 2361m 9197Mi ds-cts-0 4500m 13875Mi ds-cts-1 5448m 13843Mi ds-cts-2 2964m 13842Mi ds-idrepo-0 1824m 13823Mi ds-idrepo-1 13m 13825Mi ds-idrepo-2 15m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 5m 3278Mi idm-78f4b47cb9-vh9b7 8m 3255Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1484m 580Mi 04:29:49 DEBUG --- stderr --- 04:29:49 DEBUG 04:29:53 INFO 04:29:53 INFO [loop_until]: kubectl --namespace=xlou top node 04:29:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:29:53 INFO [loop_until]: OK (rc = 0) 04:29:53 DEBUG --- stdout --- 04:29:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 3497m 22% 11383Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 4895Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 1993m 12% 11425Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 3153m 19% 11790Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 163m 1% 3031Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 88m 0% 1328Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5165Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14804Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 6063m 38% 14657Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 65m 0% 14870Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 2048m 12% 14802Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 4473m 28% 14657Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 4892m 30% 14702Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1697m 10% 2576Mi 4% 04:29:53 DEBUG --- stderr --- 04:29:53 DEBUG 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "ds-cts-2" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "ds-idrepo-2" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "0" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "ds-cts-2" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "ds-idrepo-2" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "0" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "ds-idrepo-0" 04:30:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:30:36 DEBUG }, 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "ds-idrepo-2" 04:30:36 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:30:36 DEBUG }, 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "0" 04:30:36 DEBUG "0" 04:30:36 DEBUG ] 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "ds-idrepo-0" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "ds-idrepo-2" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "0" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "1015.1666666666666" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:30:36 DEBUG "instance": "10.142.0.17:9100" 04:30:36 DEBUG }, 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:30:36 DEBUG "instance": "10.142.0.61:9100" 04:30:36 DEBUG }, 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "1015.1333333333333" 04:30:36 DEBUG "0.0016000000000000606" 04:30:36 DEBUG ] 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467036" 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG { 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG "data": { 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "data": { 04:30:36 DEBUG "instance": "10.142.0.17:9100" 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG "result": [ 04:30:36 DEBUG 04:30:36 DEBUG { 04:30:36 DEBUG 04:30:36 DEBUG "metric": { 04:30:36 DEBUG 04:30:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:30:36 DEBUG [...] 04:30:36 DEBUG }, 04:30:36 DEBUG 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "instance": "10.142.0.61:9100" 04:30:36 DEBUG 04:30:36 DEBUG }, 04:30:36 DEBUG [...] 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG 04:30:36 DEBUG "11799155.7" 04:30:36 DEBUG 04:30:36 DEBUG ] 04:30:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:30:36 DEBUG } 04:30:36 DEBUG }, 04:30:36 DEBUG ] 04:30:36 DEBUG "value": [ 04:30:36 DEBUG } 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG } 04:30:36 DEBUG "14578561024" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "2264841.2410962214" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "338.26666666666665" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "ds-cts-2" 04:30:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:30:36 DEBUG }, 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "ds-idrepo-2" 04:30:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:30:36 DEBUG }, 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "0" 04:30:36 DEBUG "0.2" 04:30:36 DEBUG ] 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "0.004526177411565702" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "instance": "10.142.0.17:9100" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "instance": "10.142.0.61:9100" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "11802378" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "1.8865242308177503" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "338.9" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "0" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "pod": "ds-idrepo-0" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "pod": "ds-idrepo-2" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "0" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467036" 04:30:36 INFO [http_cmd]: http status code OK 04:30:36 DEBUG --- status code --- 04:30:36 DEBUG http status code is 200 (expected 200) 04:30:36 DEBUG --- http response --- 04:30:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:36 DEBUG ----- output ----- 04:30:36 DEBUG { 04:30:36 DEBUG "status": "success", 04:30:36 DEBUG "data": { 04:30:36 DEBUG "resultType": "vector", 04:30:36 DEBUG "result": [ 04:30:36 DEBUG { 04:30:36 DEBUG "metric": { 04:30:36 DEBUG "instance": "10.142.0.17:9100" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG [...] 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG 04:30:36 DEBUG "instance": "10.142.0.61:9100" 04:30:36 DEBUG }, 04:30:36 DEBUG "value": [ 04:30:36 DEBUG 1684467036, 04:30:36 DEBUG "0.0006666666666666524" 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG ] 04:30:36 DEBUG } 04:30:36 DEBUG } 04:30:49 INFO 04:30:49 INFO [loop_until]: kubectl --namespace=xlou top pods 04:30:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:30:49 INFO [loop_until]: OK (rc = 0) 04:30:49 DEBUG --- stdout --- 04:30:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7500m 9183Mi am-7849cf7bdb-jcbb8 7344m 9167Mi am-7849cf7bdb-zrhp7 7614m 9201Mi ds-cts-0 10370m 13875Mi ds-cts-1 10885m 13884Mi ds-cts-2 10576m 13845Mi ds-idrepo-0 4858m 13823Mi ds-idrepo-1 13m 13825Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3272Mi idm-78f4b47cb9-vh9b7 8m 3249Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 2047m 657Mi 04:30:49 DEBUG --- stderr --- 04:30:49 DEBUG 04:30:53 INFO 04:30:53 INFO [loop_until]: kubectl --namespace=xlou top node 04:30:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:30:54 INFO [loop_until]: OK (rc = 0) 04:30:54 DEBUG --- stdout --- 04:30:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7938m 49% 11383Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 76m 0% 4888Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8417m 52% 11428Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13525m 85% 11858Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 185m 1% 3019Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 90m 0% 1327Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5163Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14804Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 11140m 70% 14694Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 65m 0% 14874Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4954m 31% 14806Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 11222m 70% 14655Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 10946m 68% 14710Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2162m 13% 2645Mi 4% 04:30:54 DEBUG --- stderr --- 04:30:54 DEBUG 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "pod": "ds-cts-2" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "pod": "ds-idrepo-2" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "pod": "ds-idrepo-0" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "pod": "ds-idrepo-2" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "pod": "ds-idrepo-0" 04:31:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:31:36 DEBUG }, 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "pod": "ds-idrepo-2" 04:31:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:31:36 DEBUG }, 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0" 04:31:36 DEBUG "1008.7333333333333" 04:31:36 DEBUG ] 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG { 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:31:36 DEBUG "data": { 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG { 04:31:36 DEBUG "result": [ 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG { 04:31:36 DEBUG "data": { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:31:36 DEBUG "result": [ 04:31:36 DEBUG }, 04:31:36 DEBUG { 04:31:36 DEBUG "value": [ 04:31:36 DEBUG "metric": { 04:31:36 DEBUG 04:31:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:31:36 DEBUG 04:31:36 DEBUG }, 04:31:36 DEBUG 04:31:36 DEBUG "value": [ 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:31:36 DEBUG 04:31:36 DEBUG }, 04:31:36 DEBUG 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:31:36 DEBUG "1008.5" 04:31:36 DEBUG }, 04:31:36 DEBUG ] 04:31:36 DEBUG "value": [ 04:31:36 DEBUG } 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG ] 04:31:36 DEBUG "0" 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "pod": "ds-cts-2" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "pod": "ds-idrepo-2" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467096" 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "instance": "10.142.0.17:9100" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "instance": "10.142.0.61:9100" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0.0015666666666665683" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "instance": "10.142.0.17:9100" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "instance": "10.142.0.61:9100" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "11544345.833333332" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "14598270976" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG { 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:36 DEBUG "data": { 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG { 04:31:36 DEBUG { 04:31:36 DEBUG "result": [ 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG { 04:31:36 DEBUG "data": { 04:31:36 DEBUG "data": { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:31:36 DEBUG "result": [ 04:31:36 DEBUG "result": [ 04:31:36 DEBUG }, 04:31:36 DEBUG { 04:31:36 DEBUG { 04:31:36 DEBUG "value": [ 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG 04:31:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:31:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:31:36 DEBUG 04:31:36 DEBUG }, 04:31:36 DEBUG }, 04:31:36 DEBUG 04:31:36 DEBUG "value": [ 04:31:36 DEBUG "value": [ 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG [...] 04:31:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:31:36 DEBUG 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 DEBUG 04:31:36 DEBUG }, 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:31:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:31:36 DEBUG "0" 04:31:36 DEBUG }, 04:31:36 DEBUG }, 04:31:36 DEBUG ] 04:31:36 DEBUG "value": [ 04:31:36 DEBUG "value": [ 04:31:36 DEBUG } 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG ] 04:31:36 DEBUG "336.8666666666667" 04:31:36 DEBUG "0.2" 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "pod": "ds-cts-2" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "pod": "ds-idrepo-2" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467096" 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "2.0989525731689276" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467096" 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "instance": "10.142.0.17:9100" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "instance": "10.142.0.61:9100" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "11547156.433333334" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "334.9" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467096" 04:31:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467096" 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "pod": "ds-idrepo-0" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "pod": "ds-idrepo-2" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:36 INFO [http_cmd]: http status code OK 04:31:36 DEBUG --- status code --- 04:31:36 DEBUG http status code is 200 (expected 200) 04:31:36 DEBUG --- http response --- 04:31:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:36 DEBUG ----- output ----- 04:31:36 DEBUG { 04:31:36 DEBUG "status": "success", 04:31:36 DEBUG "data": { 04:31:36 DEBUG "resultType": "vector", 04:31:36 DEBUG "result": [ 04:31:36 DEBUG { 04:31:36 DEBUG "metric": { 04:31:36 DEBUG "instance": "10.142.0.17:9100" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG [...] 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG 04:31:36 DEBUG "instance": "10.142.0.61:9100" 04:31:36 DEBUG }, 04:31:36 DEBUG "value": [ 04:31:36 DEBUG 1684467096, 04:31:36 DEBUG "0.000999999999999801" 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG ] 04:31:36 DEBUG } 04:31:36 DEBUG } 04:31:49 INFO 04:31:49 INFO [loop_until]: kubectl --namespace=xlou top pods 04:31:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:31:49 INFO [loop_until]: OK (rc = 0) 04:31:49 DEBUG --- stdout --- 04:31:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 6940m 9184Mi am-7849cf7bdb-jcbb8 7107m 9167Mi am-7849cf7bdb-zrhp7 7485m 9202Mi ds-cts-0 10203m 13833Mi ds-cts-1 9911m 13883Mi ds-cts-2 9866m 13848Mi ds-idrepo-0 4571m 13823Mi ds-idrepo-1 13m 13825Mi ds-idrepo-2 14m 13827Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 11m 3249Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 2046m 667Mi 04:31:49 DEBUG --- stderr --- 04:31:49 DEBUG 04:31:54 INFO 04:31:54 INFO [loop_until]: kubectl --namespace=xlou top node 04:31:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:31:54 INFO [loop_until]: OK (rc = 0) 04:31:54 DEBUG --- stdout --- 04:31:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7903m 49% 11373Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 84m 0% 4892Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 7669m 48% 11430Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13185m 82% 11810Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 168m 1% 3026Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 91m 0% 1331Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5165Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 69m 0% 14802Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 10165m 63% 14693Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 66m 0% 14870Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4583m 28% 14803Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 9833m 61% 14660Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 10272m 64% 14657Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 2110m 13% 2653Mi 4% 04:31:54 DEBUG --- stderr --- 04:31:54 DEBUG 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "ds-cts-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "ds-idrepo-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "0" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "ds-idrepo-0" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "ds-idrepo-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "0" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:36 DEBUG { 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG { 04:32:36 DEBUG { 04:32:36 DEBUG { 04:32:36 DEBUG "data": { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "data": { 04:32:36 DEBUG "data": { 04:32:36 DEBUG "result": [ 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG { 04:32:36 DEBUG "result": [ 04:32:36 DEBUG "result": [ 04:32:36 DEBUG "result": [ 04:32:36 DEBUG "metric": { 04:32:36 DEBUG { 04:32:36 DEBUG { 04:32:36 DEBUG { 04:32:36 DEBUG "pod": "ds-idrepo-0" 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG }, 04:32:36 DEBUG "pod": "ds-cts-2" 04:32:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:32:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:32:36 DEBUG "value": [ 04:32:36 DEBUG }, 04:32:36 DEBUG }, 04:32:36 DEBUG }, 04:32:36 DEBUG 04:32:36 DEBUG "value": [ 04:32:36 DEBUG "value": [ 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG [...] 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "ds-idrepo-2" 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG }, 04:32:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:32:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:32:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:32:36 DEBUG "value": [ 04:32:36 DEBUG }, 04:32:36 DEBUG }, 04:32:36 DEBUG }, 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG "value": [ 04:32:36 DEBUG "value": [ 04:32:36 DEBUG "0" 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG ] 04:32:36 DEBUG "0" 04:32:36 DEBUG "1016.8333333333334" 04:32:36 DEBUG "1018.4" 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG ] 04:32:36 DEBUG ] 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG ] 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "ds-cts-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "ds-idrepo-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "0" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "instance": "10.142.0.17:9100" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "instance": "10.142.0.61:9100" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "0.0018333333333326134" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "instance": "10.142.0.17:9100" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "instance": "10.142.0.61:9100" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "11799867.633333333" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG "343.6" 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG ] 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG } 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG ] 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:36 DEBUG } 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG } 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "14599065600" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:32:36 DEBUG "pod": "ds-cts-2" 04:32:36 DEBUG }, 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:32:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:32:36 DEBUG }, 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "0.2" 04:32:36 DEBUG "80201.67269609944" 04:32:36 DEBUG ] 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "ds-cts-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "ds-idrepo-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "0" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "ds-cts-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "4.159332228635666" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "instance": "10.142.0.17:9100" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "instance": "10.142.0.61:9100" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "11802346.933333334" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG [...] 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:36 DEBUG 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:32:36 DEBUG { 04:32:36 DEBUG }, 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "value": [ 04:32:36 DEBUG "data": { 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "332.8" 04:32:36 DEBUG "result": [ 04:32:36 DEBUG ] 04:32:36 DEBUG { 04:32:36 DEBUG } 04:32:36 DEBUG "metric": { 04:32:36 DEBUG ] 04:32:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:32:36 DEBUG } 04:32:36 DEBUG }, 04:32:36 DEBUG } 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "2.029139128925508" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "ds-cts-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "0" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467156" 04:32:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467156" 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "pod": "ds-idrepo-0" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "pod": "ds-idrepo-2" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "0" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:36 INFO [http_cmd]: http status code OK 04:32:36 DEBUG --- status code --- 04:32:36 DEBUG http status code is 200 (expected 200) 04:32:36 DEBUG --- http response --- 04:32:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:36 DEBUG ----- output ----- 04:32:36 DEBUG { 04:32:36 DEBUG "status": "success", 04:32:36 DEBUG "data": { 04:32:36 DEBUG "resultType": "vector", 04:32:36 DEBUG "result": [ 04:32:36 DEBUG { 04:32:36 DEBUG "metric": { 04:32:36 DEBUG "instance": "10.142.0.17:9100" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG [...] 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG 04:32:36 DEBUG "instance": "10.142.0.61:9100" 04:32:36 DEBUG }, 04:32:36 DEBUG "value": [ 04:32:36 DEBUG 1684467156, 04:32:36 DEBUG "0.0006666666666667708" 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG ] 04:32:36 DEBUG } 04:32:36 DEBUG } 04:32:49 INFO 04:32:49 INFO [loop_until]: kubectl --namespace=xlou top pods 04:32:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:32:49 INFO [loop_until]: OK (rc = 0) 04:32:49 DEBUG --- stdout --- 04:32:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7964m 9187Mi am-7849cf7bdb-jcbb8 7469m 9176Mi am-7849cf7bdb-zrhp7 7938m 9208Mi ds-cts-0 10452m 13875Mi ds-cts-1 10758m 13885Mi ds-cts-2 10878m 13826Mi ds-idrepo-0 4566m 13822Mi ds-idrepo-1 12m 13825Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 7m 3256Mi lodemon-765d5f4748-dps7q 1m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 2025m 841Mi 04:32:49 DEBUG --- stderr --- 04:32:49 DEBUG 04:32:54 INFO 04:32:54 INFO [loop_until]: kubectl --namespace=xlou top node 04:32:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:32:54 INFO [loop_until]: OK (rc = 0) 04:32:54 DEBUG --- stdout --- 04:32:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 8290m 52% 11377Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 4894Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8454m 53% 11431Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13603m 85% 11851Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 175m 1% 3082Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 91m 0% 1330Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 92m 0% 5164Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14803Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 11059m 69% 14690Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 65m 0% 14877Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4620m 29% 14802Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 11067m 69% 14564Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 11099m 69% 14706Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2129m 13% 2826Mi 4% 04:32:54 DEBUG --- stderr --- 04:32:54 DEBUG 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "pod": "ds-cts-2" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "pod": "ds-idrepo-2" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "0" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467216" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467216" 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:36 DEBUG { 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG { 04:33:36 DEBUG { 04:33:36 DEBUG "data": { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "data": { 04:33:36 DEBUG "result": [ 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG { 04:33:36 DEBUG "result": [ 04:33:36 DEBUG "result": [ 04:33:36 DEBUG "metric": { 04:33:36 DEBUG { 04:33:36 DEBUG { 04:33:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG }, 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:33:36 DEBUG "instance": "10.142.0.17:9100" 04:33:36 DEBUG "value": [ 04:33:36 DEBUG }, 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG }, 04:33:36 DEBUG 04:33:36 DEBUG "value": [ 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG { 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG 04:33:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:33:36 DEBUG 04:33:36 DEBUG "data": { 04:33:36 DEBUG 04:33:36 DEBUG }, 04:33:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "instance": "10.142.0.61:9100" 04:33:36 DEBUG "value": [ 04:33:36 DEBUG }, 04:33:36 DEBUG "result": [ 04:33:36 DEBUG }, 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG { 04:33:36 DEBUG "value": [ 04:33:36 DEBUG "1021.3333333333334" 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "metric": { 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG ] 04:33:36 DEBUG "0" 04:33:36 DEBUG "pod": "ds-idrepo-0" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 DEBUG "0.0017666666666665757" 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG }, 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 DEBUG ] 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG "value": [ 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG 04:33:36 DEBUG ] 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 DEBUG 04:33:36 DEBUG } 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG } 04:33:36 DEBUG 04:33:36 DEBUG } 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [...] 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:36 DEBUG 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG 04:33:36 DEBUG { 04:33:36 DEBUG 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "pod": "ds-idrepo-2" 04:33:36 DEBUG "data": { 04:33:36 DEBUG }, 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "value": [ 04:33:36 DEBUG "result": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG { 04:33:36 DEBUG "0" 04:33:36 DEBUG "metric": { 04:33:36 DEBUG ] 04:33:36 DEBUG "instance": "10.142.0.17:9100" 04:33:36 DEBUG } 04:33:36 DEBUG }, 04:33:36 DEBUG ] 04:33:36 DEBUG "value": [ 04:33:36 DEBUG } 04:33:36 DEBUG 04:33:36 DEBUG } 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "instance": "10.142.0.61:9100" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "11755786.133333333" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "resultType": "vector", 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG "result": [ 04:33:36 DEBUG "result": [ 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG { 04:33:36 DEBUG { 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG "pod": "ds-cts-2" 04:33:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:36 DEBUG }, 04:33:36 DEBUG }, 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG "value": [ 04:33:36 DEBUG "value": [ 04:33:36 DEBUG { 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "data": { 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG [...] 04:33:36 DEBUG [...] 04:33:36 DEBUG "result": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG { 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "metric": { 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "pod": "ds-idrepo-0" 04:33:36 DEBUG "pod": "ds-idrepo-2" 04:33:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:33:36 DEBUG }, 04:33:36 DEBUG }, 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG "value": [ 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG 04:33:36 DEBUG "0" 04:33:36 DEBUG "1021.5" 04:33:36 DEBUG 04:33:36 DEBUG ] 04:33:36 DEBUG ] 04:33:36 DEBUG [...] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 DEBUG 04:33:36 DEBUG ] 04:33:36 DEBUG ] 04:33:36 DEBUG 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 DEBUG 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 DEBUG "pod": "ds-idrepo-2" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "0" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "0" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467216" 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG --- http response --- 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG { 04:33:36 DEBUG { 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:36 DEBUG "data": { 04:33:36 DEBUG "data": { 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG { 04:33:36 DEBUG "result": [ 04:33:36 DEBUG "result": [ 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG { 04:33:36 DEBUG { 04:33:36 DEBUG "data": { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "pod": "ds-cts-2" 04:33:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:33:36 DEBUG "result": [ 04:33:36 DEBUG }, 04:33:36 DEBUG }, 04:33:36 DEBUG { 04:33:36 DEBUG "value": [ 04:33:36 DEBUG "value": [ 04:33:36 DEBUG "metric": { 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG }, 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "value": [ 04:33:36 DEBUG [...] 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG "pod": "ds-idrepo-2" 04:33:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:33:36 DEBUG 04:33:36 DEBUG }, 04:33:36 DEBUG }, 04:33:36 DEBUG 04:33:36 DEBUG "value": [ 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:33:36 DEBUG "0" 04:33:36 DEBUG "340.1666666666667" 04:33:36 DEBUG }, 04:33:36 DEBUG ] 04:33:36 DEBUG ] 04:33:36 DEBUG "value": [ 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG ] 04:33:36 DEBUG ] 04:33:36 DEBUG "14599909376" 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "0.2" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "0" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467216" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467216" 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "instance": "10.142.0.17:9100" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "instance": "10.142.0.61:9100" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "11758360.166666666" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "340.96666666666664" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "2.0468388886983617" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467216" 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "pod": "ds-idrepo-0" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "pod": "ds-idrepo-2" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "0" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "instance": "10.142.0.17:9100" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "instance": "10.142.0.61:9100" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "0.0009999999999999194" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467216" 04:33:36 INFO [http_cmd]: http status code OK 04:33:36 DEBUG --- status code --- 04:33:36 DEBUG http status code is 200 (expected 200) 04:33:36 DEBUG --- http response --- 04:33:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:36 DEBUG ----- output ----- 04:33:36 DEBUG { 04:33:36 DEBUG "status": "success", 04:33:36 DEBUG "data": { 04:33:36 DEBUG "resultType": "vector", 04:33:36 DEBUG "result": [ 04:33:36 DEBUG { 04:33:36 DEBUG "metric": { 04:33:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG [...] 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG 04:33:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:33:36 DEBUG }, 04:33:36 DEBUG "value": [ 04:33:36 DEBUG 1684467216, 04:33:36 DEBUG "0" 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG ] 04:33:36 DEBUG } 04:33:36 DEBUG } 04:33:49 INFO 04:33:49 INFO [loop_until]: kubectl --namespace=xlou top pods 04:33:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:33:49 INFO [loop_until]: OK (rc = 0) 04:33:49 DEBUG --- stdout --- 04:33:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7317m 9186Mi am-7849cf7bdb-jcbb8 7154m 9175Mi am-7849cf7bdb-zrhp7 7660m 9208Mi ds-cts-0 10223m 13836Mi ds-cts-1 10756m 13887Mi ds-cts-2 10732m 13824Mi ds-idrepo-0 4699m 13823Mi ds-idrepo-1 13m 13825Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 5m 3278Mi idm-78f4b47cb9-vh9b7 8m 3251Mi lodemon-765d5f4748-dps7q 1m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 2012m 844Mi 04:33:49 DEBUG --- stderr --- 04:33:49 DEBUG 04:33:54 INFO 04:33:54 INFO [loop_until]: kubectl --namespace=xlou top node 04:33:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:33:54 INFO [loop_until]: OK (rc = 0) 04:33:54 DEBUG --- stdout --- 04:33:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7941m 49% 11371Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4888Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8130m 51% 11431Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13310m 83% 11812Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 160m 1% 3031Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 91m 0% 1332Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 92m 0% 5172Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 69m 0% 14800Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 10550m 66% 14688Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 66m 0% 14876Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4574m 28% 14803Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 10563m 66% 14658Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 10466m 65% 14661Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2082m 13% 2831Mi 4% 04:33:54 DEBUG --- stderr --- 04:33:54 DEBUG 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "pod": "ds-cts-2" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "pod": "ds-idrepo-2" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "0" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467276" 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG { 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:36 DEBUG "data": { 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG { 04:34:36 DEBUG "result": [ 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG { 04:34:36 DEBUG "data": { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "instance": "10.142.0.17:9100" 04:34:36 DEBUG "result": [ 04:34:36 DEBUG }, 04:34:36 DEBUG { 04:34:36 DEBUG "value": [ 04:34:36 DEBUG "metric": { 04:34:36 DEBUG 04:34:36 DEBUG "instance": "10.142.0.17:9100" 04:34:36 DEBUG 04:34:36 DEBUG }, 04:34:36 DEBUG 04:34:36 DEBUG "value": [ 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG "instance": "10.142.0.61:9100" 04:34:36 DEBUG 04:34:36 DEBUG }, 04:34:36 DEBUG 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "instance": "10.142.0.61:9100" 04:34:36 DEBUG "0.0016333333333326057" 04:34:36 DEBUG }, 04:34:36 DEBUG ] 04:34:36 DEBUG "value": [ 04:34:36 DEBUG } 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG ] 04:34:36 DEBUG "11703703.899999999" 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:36 DEBUG { 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG { 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "data": { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG "resultType": "vector", 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG "data": { 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG "result": [ 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG { 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG "result": [ 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG "metric": { 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG { 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:34:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:34:36 DEBUG "metric": { 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG }, 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG "pod": "ds-idrepo-0" 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "pod": "ds-idrepo-0" 04:34:36 DEBUG { 04:34:36 DEBUG "value": [ 04:34:36 DEBUG { 04:34:36 DEBUG { 04:34:36 DEBUG }, 04:34:36 DEBUG { 04:34:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:34:36 DEBUG }, 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "value": [ 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG "data": { 04:34:36 DEBUG 04:34:36 DEBUG "data": { 04:34:36 DEBUG "data": { 04:34:36 DEBUG 04:34:36 DEBUG "data": { 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "result": [ 04:34:36 DEBUG [...] 04:34:36 DEBUG "result": [ 04:34:36 DEBUG "result": [ 04:34:36 DEBUG 04:34:36 DEBUG "result": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG { 04:34:36 DEBUG 04:34:36 DEBUG { 04:34:36 DEBUG { 04:34:36 DEBUG [...] 04:34:36 DEBUG { 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG "metric": { 04:34:36 DEBUG 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG 04:34:36 DEBUG "metric": { 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG "pod": "ds-cts-2" 04:34:36 DEBUG 04:34:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:34:36 DEBUG "pod": "ds-cts-2" 04:34:36 DEBUG 04:34:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG }, 04:34:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:34:36 DEBUG }, 04:34:36 DEBUG }, 04:34:36 DEBUG 04:34:36 DEBUG }, 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "value": [ 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG "value": [ 04:34:36 DEBUG "pod": "ds-idrepo-2" 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG "pod": "ds-idrepo-2" 04:34:36 DEBUG 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG }, 04:34:36 DEBUG 04:34:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:34:36 DEBUG }, 04:34:36 DEBUG 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG "0" 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG [...] 04:34:36 DEBUG ] 04:34:36 DEBUG [...] 04:34:36 DEBUG [...] 04:34:36 DEBUG "0" 04:34:36 DEBUG [...] 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "0" 04:34:36 DEBUG 04:34:36 DEBUG } 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG ] 04:34:36 DEBUG 04:34:36 DEBUG "1042.5666666666666" 04:34:36 DEBUG ] 04:34:36 DEBUG 04:34:36 DEBUG ] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG } 04:34:36 DEBUG 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG 04:34:36 DEBUG } 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG ] 04:34:36 DEBUG 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG "pod": "ds-idrepo-2" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 DEBUG } 04:34:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:34:36 DEBUG "pod": "ds-idrepo-2" 04:34:36 DEBUG } 04:34:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG }, 04:34:36 DEBUG }, 04:34:36 DEBUG }, 04:34:36 DEBUG } 04:34:36 DEBUG }, 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG "value": [ 04:34:36 DEBUG "value": [ 04:34:36 DEBUG "value": [ 04:34:36 DEBUG "value": [ 04:34:36 DEBUG } 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "0" 04:34:36 DEBUG "347.43333333333334" 04:34:36 DEBUG "0" 04:34:36 DEBUG "1042.5333333333333" 04:34:36 DEBUG ] 04:34:36 DEBUG ] 04:34:36 DEBUG ] 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG ] 04:34:36 DEBUG ] 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "pod": "end-user-ui-787cb4f6b4-56bl4" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "0" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467276" 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "0.2" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "14598549504" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467276" 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "2.0396953774804483" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "pod": "end-user-ui-787cb4f6b4-56bl4" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "0" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "instance": "10.142.0.17:9100" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "instance": "10.142.0.61:9100" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "11704464.3" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "347.5333333333333" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467276" 04:34:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467276" 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG { 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG "data": { 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "data": { 04:34:36 DEBUG "pod": "ds-idrepo-0" 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG }, 04:34:36 DEBUG "result": [ 04:34:36 DEBUG "value": [ 04:34:36 DEBUG { 04:34:36 DEBUG 04:34:36 DEBUG "metric": { 04:34:36 DEBUG 04:34:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:34:36 DEBUG 04:34:36 DEBUG }, 04:34:36 DEBUG [...] 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "pod": "ds-idrepo-2" 04:34:36 DEBUG [...] 04:34:36 DEBUG }, 04:34:36 DEBUG 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG 04:34:36 DEBUG "0" 04:34:36 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:34:36 DEBUG ] 04:34:36 DEBUG }, 04:34:36 DEBUG } 04:34:36 DEBUG "value": [ 04:34:36 DEBUG ] 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG } 04:34:36 DEBUG "0" 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:36 INFO [http_cmd]: http status code OK 04:34:36 DEBUG --- status code --- 04:34:36 DEBUG http status code is 200 (expected 200) 04:34:36 DEBUG --- http response --- 04:34:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:36 DEBUG ----- output ----- 04:34:36 DEBUG { 04:34:36 DEBUG "status": "success", 04:34:36 DEBUG "data": { 04:34:36 DEBUG "resultType": "vector", 04:34:36 DEBUG "result": [ 04:34:36 DEBUG { 04:34:36 DEBUG "metric": { 04:34:36 DEBUG "instance": "10.142.0.17:9100" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG [...] 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG 04:34:36 DEBUG "instance": "10.142.0.61:9100" 04:34:36 DEBUG }, 04:34:36 DEBUG "value": [ 04:34:36 DEBUG 1684467276, 04:34:36 DEBUG "0.00066666666666683" 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG ] 04:34:36 DEBUG } 04:34:36 DEBUG } 04:34:49 INFO 04:34:49 INFO [loop_until]: kubectl --namespace=xlou top pods 04:34:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:34:49 INFO [loop_until]: OK (rc = 0) 04:34:49 DEBUG --- stdout --- 04:34:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7425m 9189Mi am-7849cf7bdb-jcbb8 7335m 9176Mi am-7849cf7bdb-zrhp7 7534m 9208Mi ds-cts-0 10372m 13876Mi ds-cts-1 10481m 13883Mi ds-cts-2 10814m 13880Mi ds-idrepo-0 4981m 13823Mi ds-idrepo-1 14m 13825Mi ds-idrepo-2 15m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 5m 3271Mi idm-78f4b47cb9-vh9b7 8m 3257Mi lodemon-765d5f4748-dps7q 4m 68Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1982m 847Mi 04:34:49 DEBUG --- stderr --- 04:34:49 DEBUG 04:34:54 INFO 04:34:54 INFO [loop_until]: kubectl --namespace=xlou top node 04:34:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:34:54 INFO [loop_until]: OK (rc = 0) 04:34:54 DEBUG --- stdout --- 04:34:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7927m 49% 11372Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 82m 0% 4896Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 7967m 50% 11432Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13212m 83% 11853Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 162m 1% 3080Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 90m 0% 1330Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 88m 0% 5165Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 70m 0% 14801Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 10655m 67% 14690Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 66m 0% 14875Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4977m 31% 14805Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 10699m 67% 14682Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 10519m 66% 14635Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 2032m 12% 2832Mi 4% 04:34:54 DEBUG --- stderr --- 04:34:54 DEBUG 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "pod": "ds-cts-2" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "pod": "ds-idrepo-2" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "0" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "instance": "10.142.0.17:9100" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "instance": "10.142.0.61:9100" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "0.0016333333333326057" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "instance": "10.142.0.17:9100" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "instance": "10.142.0.61:9100" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "11823226.533333333" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "11534.896345808027" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG --- status code --- 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG { 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG "status": "success", 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG ----- output ----- 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:35:36 DEBUG "data": { 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG { 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG { 04:35:36 DEBUG "result": [ 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG "data": { 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG { 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:36 DEBUG "data": { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG "result": [ 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG "resultType": "vector", 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG "pod": "ds-cts-2" 04:35:36 DEBUG { 04:35:36 DEBUG { 04:35:36 DEBUG { 04:35:36 DEBUG { 04:35:36 DEBUG "result": [ 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG }, 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG { 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG "value": [ 04:35:36 DEBUG "data": { 04:35:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:35:36 DEBUG "data": { 04:35:36 DEBUG "data": { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG }, 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "pod": "ds-cts-2" 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:36 DEBUG 04:35:36 DEBUG "result": [ 04:35:36 DEBUG "value": [ 04:35:36 DEBUG "result": [ 04:35:36 DEBUG "result": [ 04:35:36 DEBUG }, 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG 04:35:36 DEBUG { 04:35:36 DEBUG 04:35:36 DEBUG { 04:35:36 DEBUG { 04:35:36 DEBUG "value": [ 04:35:36 DEBUG { 04:35:36 DEBUG { 04:35:36 DEBUG [...] 04:35:36 DEBUG "metric": { 04:35:36 DEBUG 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG 04:35:36 DEBUG "pod": "ds-idrepo-0" 04:35:36 DEBUG 04:35:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:35:36 DEBUG "pod": "ds-idrepo-0" 04:35:36 DEBUG 04:35:36 DEBUG "data": { 04:35:36 DEBUG "data": { 04:35:36 DEBUG 04:35:36 DEBUG }, 04:35:36 DEBUG [...] 04:35:36 DEBUG }, 04:35:36 DEBUG }, 04:35:36 DEBUG 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG "value": [ 04:35:36 DEBUG "value": [ 04:35:36 DEBUG [...] 04:35:36 DEBUG "result": [ 04:35:36 DEBUG "result": [ 04:35:36 DEBUG "pod": "ds-idrepo-2" 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG { 04:35:36 DEBUG { 04:35:36 DEBUG }, 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:35:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG [...] 04:35:36 DEBUG }, 04:35:36 DEBUG [...] 04:35:36 DEBUG [...] 04:35:36 DEBUG "pod": "ds-idrepo-2" 04:35:36 DEBUG }, 04:35:36 DEBUG }, 04:35:36 DEBUG "0" 04:35:36 DEBUG 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG "value": [ 04:35:36 DEBUG ] 04:35:36 DEBUG 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG } 04:35:36 DEBUG 04:35:36 DEBUG "0" 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG ] 04:35:36 DEBUG "pod": "ds-idrepo-2" 04:35:36 DEBUG ] 04:35:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:35:36 DEBUG "pod": "ds-idrepo-2" 04:35:36 DEBUG "0" 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG } 04:35:36 DEBUG }, 04:35:36 DEBUG } 04:35:36 DEBUG }, 04:35:36 DEBUG }, 04:35:36 DEBUG ] 04:35:36 DEBUG [...] 04:35:36 DEBUG [...] 04:35:36 DEBUG } 04:35:36 DEBUG "value": [ 04:35:36 DEBUG ] 04:35:36 DEBUG "value": [ 04:35:36 DEBUG "value": [ 04:35:36 DEBUG } 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG } 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG ] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "0" 04:35:36 DEBUG } 04:35:36 DEBUG "1034.6333333333334" 04:35:36 DEBUG "0" 04:35:36 DEBUG } 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG ] 04:35:36 DEBUG ] 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:35:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG }, 04:35:36 DEBUG }, 04:35:36 DEBUG ] 04:35:36 DEBUG ] 04:35:36 DEBUG ] 04:35:36 DEBUG "value": [ 04:35:36 DEBUG "value": [ 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG "344.96666666666664" 04:35:36 DEBUG "1034.7" 04:35:36 DEBUG ] 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467336" 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "0.2" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "14600269824" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "1.9774013623164164" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "7.113029319445063" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "344.73333333333335" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "instance": "10.142.0.17:9100" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "instance": "10.142.0.61:9100" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "11825888.4" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467336" 04:35:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467336" 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:35:36 DEBUG "pod": "ds-idrepo-0" 04:35:36 DEBUG }, 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:35:36 DEBUG "pod": "ds-idrepo-2" 04:35:36 DEBUG }, 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "0" 04:35:36 DEBUG "0" 04:35:36 DEBUG ] 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:36 INFO [http_cmd]: http status code OK 04:35:36 DEBUG --- status code --- 04:35:36 DEBUG http status code is 200 (expected 200) 04:35:36 DEBUG --- http response --- 04:35:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:36 DEBUG ----- output ----- 04:35:36 DEBUG { 04:35:36 DEBUG "status": "success", 04:35:36 DEBUG "data": { 04:35:36 DEBUG "resultType": "vector", 04:35:36 DEBUG "result": [ 04:35:36 DEBUG { 04:35:36 DEBUG "metric": { 04:35:36 DEBUG "instance": "10.142.0.17:9100" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG [...] 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG 04:35:36 DEBUG "instance": "10.142.0.61:9100" 04:35:36 DEBUG }, 04:35:36 DEBUG "value": [ 04:35:36 DEBUG 1684467336, 04:35:36 DEBUG "0.0009999999999996826" 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG ] 04:35:36 DEBUG } 04:35:36 DEBUG } 04:35:49 INFO 04:35:49 INFO [loop_until]: kubectl --namespace=xlou top pods 04:35:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:35:50 INFO [loop_until]: OK (rc = 0) 04:35:50 DEBUG --- stdout --- 04:35:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7253m 9189Mi am-7849cf7bdb-jcbb8 7260m 9177Mi am-7849cf7bdb-zrhp7 7386m 9209Mi ds-cts-0 10791m 13872Mi ds-cts-1 10572m 13852Mi ds-cts-2 10351m 13848Mi ds-idrepo-0 4998m 13822Mi ds-idrepo-1 13m 13826Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3271Mi idm-78f4b47cb9-vh9b7 8m 3257Mi lodemon-765d5f4748-dps7q 3m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1995m 848Mi 04:35:50 DEBUG --- stderr --- 04:35:50 DEBUG 04:35:54 INFO 04:35:54 INFO [loop_until]: kubectl --namespace=xlou top node 04:35:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:35:54 INFO [loop_until]: OK (rc = 0) 04:35:54 DEBUG --- stdout --- 04:35:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 8047m 50% 11372Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4896Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8194m 51% 11439Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13282m 83% 11813Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 160m 1% 3090Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 91m 0% 1333Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5165Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14803Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 10762m 67% 14628Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 65m 0% 14875Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 5019m 31% 14806Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 10602m 66% 14654Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 10248m 64% 14701Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2038m 12% 2830Mi 4% 04:35:54 DEBUG --- stderr --- 04:35:54 DEBUG 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "pod": "ds-cts-2" 04:36:36 DEBUG "instance": "10.142.0.17:9100" 04:36:36 DEBUG }, 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "pod": "ds-idrepo-2" 04:36:36 DEBUG "instance": "10.142.0.61:9100" 04:36:36 DEBUG }, 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "0" 04:36:36 DEBUG "0.0018333333333326134" 04:36:36 DEBUG ] 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "instance": "10.142.0.17:9100" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "instance": "10.142.0.61:9100" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "11711473.3" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "1030.6" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "0" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "pod": "ds-idrepo-0" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "pod": "ds-idrepo-2" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "0" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG { 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG "result": [ 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG { 04:36:36 DEBUG { 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:36:36 DEBUG "pod": "ds-idrepo-0" 04:36:36 DEBUG "data": { 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG }, 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG { 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG { 04:36:36 DEBUG "value": [ 04:36:36 DEBUG "result": [ 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG { 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG 04:36:36 DEBUG { 04:36:36 DEBUG "data": { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG "data": { 04:36:36 DEBUG 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "data": { 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG 04:36:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:36:36 DEBUG "result": [ 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG "result": [ 04:36:36 DEBUG [...] 04:36:36 DEBUG }, 04:36:36 DEBUG { 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG { 04:36:36 DEBUG 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG "value": [ 04:36:36 DEBUG "metric": { 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "metric": { 04:36:36 DEBUG 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG 04:36:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "data": { 04:36:36 DEBUG "pod": "ds-cts-2" 04:36:36 DEBUG 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG 04:36:36 DEBUG }, 04:36:36 DEBUG "pod": "ds-cts-2" 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG }, 04:36:36 DEBUG "pod": "ds-idrepo-2" 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG 04:36:36 DEBUG "value": [ 04:36:36 DEBUG }, 04:36:36 DEBUG "result": [ 04:36:36 DEBUG "value": [ 04:36:36 DEBUG }, 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG "value": [ 04:36:36 DEBUG { 04:36:36 DEBUG 04:36:36 DEBUG "value": [ 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "metric": { 04:36:36 DEBUG 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG { 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:36:36 DEBUG 04:36:36 DEBUG "0" 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG }, 04:36:36 DEBUG [...] 04:36:36 DEBUG ] 04:36:36 DEBUG "data": { 04:36:36 DEBUG "pod": "ds-cts-1" 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG } 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG }, 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG ] 04:36:36 DEBUG "result": [ 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG } 04:36:36 DEBUG { 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "pod": "ds-idrepo-2" 04:36:36 DEBUG } 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "489.49875770224605" 04:36:36 DEBUG }, 04:36:36 DEBUG "pod": "ds-idrepo-2" 04:36:36 DEBUG [...] 04:36:36 DEBUG }, 04:36:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:36:36 DEBUG ] 04:36:36 DEBUG "value": [ 04:36:36 DEBUG }, 04:36:36 DEBUG 04:36:36 DEBUG "value": [ 04:36:36 DEBUG }, 04:36:36 DEBUG } 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG ] 04:36:36 DEBUG "1030.4666666666667" 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG 04:36:36 DEBUG "0" 04:36:36 DEBUG 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG "0" 04:36:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:36:36 DEBUG ] 04:36:36 DEBUG 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG }, 04:36:36 DEBUG } 04:36:36 DEBUG 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG "value": [ 04:36:36 DEBUG ] 04:36:36 DEBUG [...] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG } 04:36:36 DEBUG 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 DEBUG "0.2" 04:36:36 DEBUG } 04:36:36 DEBUG 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG 04:36:36 DEBUG } 04:36:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:36:36 DEBUG ] 04:36:36 DEBUG }, 04:36:36 DEBUG } 04:36:36 DEBUG "value": [ 04:36:36 DEBUG } 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "342.8" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "14602780672" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "2.000341373571991" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "0" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG { 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG "data": { 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "data": { 04:36:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG }, 04:36:36 DEBUG "result": [ 04:36:36 DEBUG "value": [ 04:36:36 DEBUG { 04:36:36 DEBUG 04:36:36 DEBUG "metric": { 04:36:36 DEBUG 04:36:36 DEBUG "instance": "10.142.0.17:9100" 04:36:36 DEBUG 04:36:36 DEBUG }, 04:36:36 DEBUG [...] 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:36:36 DEBUG [...] 04:36:36 DEBUG }, 04:36:36 DEBUG 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG 04:36:36 DEBUG "344.43333333333334" 04:36:36 DEBUG "instance": "10.142.0.61:9100" 04:36:36 DEBUG ] 04:36:36 DEBUG }, 04:36:36 DEBUG } 04:36:36 DEBUG "value": [ 04:36:36 DEBUG ] 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG } 04:36:36 DEBUG "11713823.633333333" 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 DEBUG "metric": { 04:36:36 DEBUG "instance": "10.142.0.17:9100" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG [...] 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "instance": "10.142.0.61:9100" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "0.0006666666666666524" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467396" 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG { 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG "data": { 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG "result": [ 04:36:36 DEBUG { 04:36:36 INFO [http_cmd]: http status code OK 04:36:36 DEBUG "metric": { 04:36:36 DEBUG --- status code --- 04:36:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:36:36 DEBUG http status code is 200 (expected 200) 04:36:36 DEBUG }, 04:36:36 DEBUG --- http response --- 04:36:36 DEBUG "value": [ 04:36:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:36 DEBUG 04:36:36 DEBUG ----- output ----- 04:36:36 DEBUG 04:36:36 DEBUG { 04:36:36 DEBUG 04:36:36 DEBUG "status": "success", 04:36:36 DEBUG [...] 04:36:36 DEBUG "data": { 04:36:36 DEBUG 04:36:36 DEBUG "resultType": "vector", 04:36:36 DEBUG 04:36:36 DEBUG "result": [ 04:36:36 DEBUG 04:36:36 DEBUG { 04:36:36 DEBUG "pod": "ds-cts-1" 04:36:36 DEBUG "metric": { 04:36:36 DEBUG }, 04:36:36 DEBUG "pod": "ds-idrepo-0" 04:36:36 DEBUG "value": [ 04:36:36 DEBUG }, 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG "33.546387398131586" 04:36:36 DEBUG 04:36:36 DEBUG ] 04:36:36 DEBUG 04:36:36 DEBUG } 04:36:36 DEBUG 04:36:36 DEBUG ] 04:36:36 DEBUG [...] 04:36:36 DEBUG } 04:36:36 DEBUG 04:36:36 DEBUG } 04:36:36 DEBUG 04:36:36 DEBUG 04:36:36 DEBUG "pod": "ds-idrepo-2" 04:36:36 DEBUG }, 04:36:36 DEBUG "value": [ 04:36:36 DEBUG 1684467396, 04:36:36 DEBUG "0" 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG ] 04:36:36 DEBUG } 04:36:36 DEBUG } 04:36:50 INFO 04:36:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:36:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:36:50 INFO [loop_until]: OK (rc = 0) 04:36:50 DEBUG --- stdout --- 04:36:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7375m 9190Mi am-7849cf7bdb-jcbb8 6760m 9178Mi am-7849cf7bdb-zrhp7 7622m 9316Mi ds-cts-0 10736m 13879Mi ds-cts-1 10623m 13887Mi ds-cts-2 10726m 13849Mi ds-idrepo-0 4114m 13823Mi ds-idrepo-1 13m 13825Mi ds-idrepo-2 15m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3271Mi idm-78f4b47cb9-vh9b7 8m 3257Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 2033m 848Mi 04:36:50 DEBUG --- stderr --- 04:36:50 DEBUG 04:36:54 INFO 04:36:54 INFO [loop_until]: kubectl --namespace=xlou top node 04:36:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:36:54 INFO [loop_until]: OK (rc = 0) 04:36:54 DEBUG --- stdout --- 04:36:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7859m 49% 11373Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4894Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8162m 51% 11436Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 12857m 80% 11916Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 165m 1% 3048Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 92m 0% 1329Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 87m 0% 5167Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14802Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 10582m 66% 14692Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 66m 0% 14875Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4196m 26% 14803Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 10227m 64% 14654Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 10420m 65% 14699Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2085m 13% 2835Mi 4% 04:36:54 DEBUG --- stderr --- 04:36:54 DEBUG 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467456" 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "instance": "10.142.0.17:9100" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "instance": "10.142.0.61:9100" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "0.0018666666666661057" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "pod": "ds-cts-2" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "ds-idrepo-2" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "0" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "316307405.65201694" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467456" 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "instance": "10.142.0.17:9100" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "instance": "10.142.0.61:9100" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "11866879.833333332" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "1017" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "1016.5" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "pod": "ds-idrepo-0" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "ds-idrepo-2" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "0" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "pod": "ds-idrepo-0" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "ds-idrepo-2" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "0" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467456" 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG ----- output ----- 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:36 DEBUG { 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG { 04:37:36 DEBUG "data": { 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467456" 04:37:36 DEBUG --- http response --- 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG "data": { 04:37:36 DEBUG "result": [ 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG { 04:37:36 DEBUG { 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG "result": [ 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:37:36 DEBUG { 04:37:36 DEBUG "pod": "ds-cts-2" 04:37:36 DEBUG "data": { 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG "metric": { 04:37:36 DEBUG }, 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG { 04:37:36 DEBUG { 04:37:36 DEBUG { 04:37:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:37:36 DEBUG "value": [ 04:37:36 DEBUG "result": [ 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG }, 04:37:36 DEBUG 04:37:36 DEBUG { 04:37:36 DEBUG "data": { 04:37:36 DEBUG "data": { 04:37:36 DEBUG "data": { 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:37:36 DEBUG "result": [ 04:37:36 DEBUG "result": [ 04:37:36 DEBUG "result": [ 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG }, 04:37:36 DEBUG { 04:37:36 DEBUG { 04:37:36 DEBUG { 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "value": [ 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:37:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:37:36 DEBUG "pod": "ds-cts-2" 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG }, 04:37:36 DEBUG }, 04:37:36 DEBUG }, 04:37:36 DEBUG 04:37:36 DEBUG "pod": "ds-idrepo-2" 04:37:36 DEBUG 04:37:36 DEBUG "value": [ 04:37:36 DEBUG "value": [ 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG }, 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG }, 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "value": [ 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG "0" 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG [...] 04:37:36 DEBUG [...] 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG ] 04:37:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "0" 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG } 04:37:36 DEBUG }, 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG ] 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG ] 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG } 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:36 DEBUG } 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:37:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:37:36 DEBUG "pod": "ds-idrepo-2" 04:37:36 DEBUG ] 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG } 04:37:36 DEBUG "1.992681524880688" 04:37:36 DEBUG }, 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG } 04:37:36 DEBUG { 04:37:36 DEBUG ] 04:37:36 DEBUG "value": [ 04:37:36 DEBUG }, 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG } 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG } 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG "0.2" 04:37:36 DEBUG "data": { 04:37:36 DEBUG ] 04:37:36 DEBUG "337.7" 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG ] 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG "0" 04:37:36 DEBUG } 04:37:36 DEBUG "result": [ 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG ] 04:37:36 DEBUG { 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 DEBUG "metric": { 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 DEBUG }, 04:37:36 DEBUG } 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "14599884800" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467456" 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "341.2" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "7.128199812518468" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467456" 04:37:36 INFO [http_cmd]: http status code OK 04:37:36 DEBUG --- status code --- 04:37:36 DEBUG http status code is 200 (expected 200) 04:37:36 DEBUG --- http response --- 04:37:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:36 DEBUG ----- output ----- 04:37:36 DEBUG { 04:37:36 DEBUG "status": "success", 04:37:36 DEBUG "data": { 04:37:36 DEBUG "resultType": "vector", 04:37:36 DEBUG "result": [ 04:37:36 DEBUG { 04:37:36 DEBUG "metric": { 04:37:36 DEBUG "instance": "10.142.0.17:9100" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG [...] 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG 04:37:36 DEBUG "instance": "10.142.0.61:9100" 04:37:36 DEBUG }, 04:37:36 DEBUG "value": [ 04:37:36 DEBUG 1684467456, 04:37:36 DEBUG "11869507.733333334" 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG ] 04:37:36 DEBUG } 04:37:36 DEBUG } 04:37:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467457" 04:37:37 INFO [http_cmd]: http status code OK 04:37:37 DEBUG --- status code --- 04:37:37 DEBUG http status code is 200 (expected 200) 04:37:37 DEBUG --- http response --- 04:37:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:37 DEBUG ----- output ----- 04:37:37 DEBUG { 04:37:37 DEBUG "status": "success", 04:37:37 DEBUG "data": { 04:37:37 DEBUG "resultType": "vector", 04:37:37 DEBUG "result": [ 04:37:37 DEBUG { 04:37:37 DEBUG "metric": { 04:37:37 DEBUG "instance": "10.142.0.17:9100" 04:37:37 DEBUG }, 04:37:37 DEBUG "value": [ 04:37:37 DEBUG 04:37:37 DEBUG 04:37:37 DEBUG 04:37:37 DEBUG [...] 04:37:37 DEBUG 04:37:37 DEBUG 04:37:37 DEBUG 04:37:37 DEBUG "instance": "10.142.0.61:9100" 04:37:37 DEBUG }, 04:37:37 DEBUG "value": [ 04:37:37 DEBUG 1684467457, 04:37:37 DEBUG "0.0013333333333335417" 04:37:37 DEBUG ] 04:37:37 DEBUG } 04:37:37 DEBUG ] 04:37:37 DEBUG } 04:37:37 DEBUG } 04:37:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467457" 04:37:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467457" 04:37:37 INFO [http_cmd]: http status code OK 04:37:37 DEBUG --- status code --- 04:37:37 DEBUG http status code is 200 (expected 200) 04:37:37 DEBUG --- http response --- 04:37:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:37 DEBUG ----- output ----- 04:37:37 INFO [http_cmd]: http status code OK 04:37:37 DEBUG { 04:37:37 DEBUG --- status code --- 04:37:37 DEBUG "status": "success", 04:37:37 DEBUG http status code is 200 (expected 200) 04:37:37 DEBUG "data": { 04:37:37 DEBUG --- http response --- 04:37:37 DEBUG "resultType": "vector", 04:37:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:37 DEBUG "result": [ 04:37:37 DEBUG ----- output ----- 04:37:37 DEBUG { 04:37:37 DEBUG { 04:37:37 DEBUG "metric": { 04:37:37 DEBUG "status": "success", 04:37:37 DEBUG "pod": "ds-idrepo-0" 04:37:37 DEBUG "data": { 04:37:37 DEBUG }, 04:37:37 DEBUG "resultType": "vector", 04:37:37 DEBUG "value": [ 04:37:37 DEBUG "result": [ 04:37:37 DEBUG 04:37:37 DEBUG { 04:37:37 DEBUG 04:37:37 DEBUG "metric": { 04:37:37 DEBUG 04:37:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:37:37 DEBUG [...] 04:37:37 DEBUG }, 04:37:37 DEBUG 04:37:37 DEBUG "value": [ 04:37:37 DEBUG 04:37:37 DEBUG 04:37:37 DEBUG 04:37:37 DEBUG 04:37:37 DEBUG "pod": "ds-idrepo-2" 04:37:37 DEBUG 04:37:37 DEBUG }, 04:37:37 DEBUG [...] 04:37:37 DEBUG "value": [ 04:37:37 DEBUG 04:37:37 DEBUG 1684467457, 04:37:37 DEBUG 04:37:37 DEBUG "0" 04:37:37 DEBUG 04:37:37 DEBUG ] 04:37:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:37:37 DEBUG } 04:37:37 DEBUG }, 04:37:37 DEBUG ] 04:37:37 DEBUG "value": [ 04:37:37 DEBUG } 04:37:37 DEBUG 1684467457, 04:37:37 DEBUG } 04:37:37 DEBUG "0" 04:37:37 DEBUG ] 04:37:37 DEBUG } 04:37:37 DEBUG ] 04:37:37 DEBUG } 04:37:37 DEBUG } 04:37:50 INFO 04:37:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:37:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:37:50 INFO [loop_until]: OK (rc = 0) 04:37:50 DEBUG --- stdout --- 04:37:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7270m 9190Mi am-7849cf7bdb-jcbb8 7145m 9178Mi am-7849cf7bdb-zrhp7 7472m 9210Mi ds-cts-0 10576m 13874Mi ds-cts-1 10488m 13885Mi ds-cts-2 10708m 13849Mi ds-idrepo-0 4328m 13822Mi ds-idrepo-1 13m 13825Mi ds-idrepo-2 15m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3277Mi idm-78f4b47cb9-vh9b7 7m 3258Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1970m 850Mi 04:37:50 DEBUG --- stderr --- 04:37:50 DEBUG 04:37:55 INFO 04:37:55 INFO [loop_until]: kubectl --namespace=xlou top node 04:37:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:37:55 INFO [loop_until]: OK (rc = 0) 04:37:55 DEBUG --- stdout --- 04:37:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7762m 48% 11370Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 76m 0% 4894Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8164m 51% 11434Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13357m 84% 11839Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 158m 0% 3087Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 90m 0% 1330Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 84m 0% 5169Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14801Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 10723m 67% 14700Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 62m 0% 14874Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4479m 28% 14805Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 10675m 67% 14666Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 10612m 66% 14718Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1984m 12% 2832Mi 4% 04:37:55 DEBUG --- stderr --- 04:37:55 DEBUG 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467516" 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "pod": "ds-cts-2" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "pod": "ds-idrepo-2" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "0" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "instance": "10.142.0.17:9100" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "instance": "10.142.0.61:9100" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "0.0014666666666660908" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "90524.31701719639" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "1011.4666666666667" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467516" 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "instance": "10.142.0.17:9100" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "instance": "10.142.0.61:9100" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "11801207.133333333" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "pod": "ds-idrepo-0" 04:38:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:38:36 DEBUG }, 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "pod": "ds-idrepo-2" 04:38:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:38:36 DEBUG }, 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "0" 04:38:36 DEBUG "1011.1333333333333" 04:38:36 DEBUG ] 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "pod": "ds-idrepo-0" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "pod": "ds-idrepo-2" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "0" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "14600323072" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "1.9896063021987163" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "333.8" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467516" 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:36 DEBUG { 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG { 04:38:36 DEBUG { 04:38:36 DEBUG "data": { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "data": { 04:38:36 DEBUG "result": [ 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG { 04:38:36 DEBUG "result": [ 04:38:36 DEBUG "result": [ 04:38:36 DEBUG "metric": { 04:38:36 DEBUG { 04:38:36 DEBUG { 04:38:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG }, 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:38:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:38:36 DEBUG "value": [ 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG }, 04:38:36 DEBUG }, 04:38:36 DEBUG 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG "value": [ 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG { 04:38:36 DEBUG [...] 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "data": { 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG }, 04:38:36 DEBUG "result": [ 04:38:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:38:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:38:36 DEBUG "value": [ 04:38:36 DEBUG { 04:38:36 DEBUG }, 04:38:36 DEBUG }, 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "value": [ 04:38:36 DEBUG "value": [ 04:38:36 DEBUG "0.2" 04:38:36 DEBUG "pod": "ds-cts-2" 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG ] 04:38:36 DEBUG }, 04:38:36 DEBUG "0" 04:38:36 DEBUG "338.96666666666664" 04:38:36 DEBUG } 04:38:36 DEBUG "value": [ 04:38:36 DEBUG ] 04:38:36 DEBUG ] 04:38:36 DEBUG ] 04:38:36 DEBUG 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 DEBUG 04:38:36 DEBUG ] 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 DEBUG [...] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "pod": "ds-idrepo-2" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "0" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG { 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG "data": { 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "data": { 04:38:36 DEBUG "pod": "ds-cts-2" 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG }, 04:38:36 DEBUG "result": [ 04:38:36 DEBUG "value": [ 04:38:36 DEBUG { 04:38:36 DEBUG 04:38:36 DEBUG "metric": { 04:38:36 DEBUG 04:38:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:38:36 DEBUG 04:38:36 DEBUG }, 04:38:36 DEBUG [...] 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "pod": "ds-idrepo-2" 04:38:36 DEBUG [...] 04:38:36 DEBUG }, 04:38:36 DEBUG 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG 04:38:36 DEBUG "0" 04:38:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:38:36 DEBUG ] 04:38:36 DEBUG }, 04:38:36 DEBUG } 04:38:36 DEBUG "value": [ 04:38:36 DEBUG ] 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG } 04:38:36 DEBUG "7.324955069854644" 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467516" 04:38:36 INFO [http_cmd]: http status code OK 04:38:36 DEBUG --- status code --- 04:38:36 DEBUG http status code is 200 (expected 200) 04:38:36 DEBUG --- http response --- 04:38:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:36 DEBUG ----- output ----- 04:38:36 DEBUG { 04:38:36 DEBUG "status": "success", 04:38:36 DEBUG "data": { 04:38:36 DEBUG "resultType": "vector", 04:38:36 DEBUG "result": [ 04:38:36 DEBUG { 04:38:36 DEBUG "metric": { 04:38:36 DEBUG "instance": "10.142.0.17:9100" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG [...] 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG 04:38:36 DEBUG "instance": "10.142.0.61:9100" 04:38:36 DEBUG }, 04:38:36 DEBUG "value": [ 04:38:36 DEBUG 1684467516, 04:38:36 DEBUG "11803810.233333334" 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG ] 04:38:36 DEBUG } 04:38:36 DEBUG } 04:38:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467517" 04:38:37 INFO [http_cmd]: http status code OK 04:38:37 DEBUG --- status code --- 04:38:37 DEBUG http status code is 200 (expected 200) 04:38:37 DEBUG --- http response --- 04:38:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:37 DEBUG ----- output ----- 04:38:37 DEBUG { 04:38:37 DEBUG "status": "success", 04:38:37 DEBUG "data": { 04:38:37 DEBUG "resultType": "vector", 04:38:37 DEBUG "result": [ 04:38:37 DEBUG { 04:38:37 DEBUG "metric": { 04:38:37 DEBUG "instance": "10.142.0.17:9100" 04:38:37 DEBUG }, 04:38:37 DEBUG "value": [ 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG [...] 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG "instance": "10.142.0.61:9100" 04:38:37 DEBUG }, 04:38:37 DEBUG "value": [ 04:38:37 DEBUG 1684467517, 04:38:37 DEBUG "0.001000000000000334" 04:38:37 DEBUG ] 04:38:37 DEBUG } 04:38:37 DEBUG ] 04:38:37 DEBUG } 04:38:37 DEBUG } 04:38:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467517" 04:38:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467517" 04:38:37 INFO [http_cmd]: http status code OK 04:38:37 DEBUG --- status code --- 04:38:37 DEBUG http status code is 200 (expected 200) 04:38:37 DEBUG --- http response --- 04:38:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:37 DEBUG ----- output ----- 04:38:37 DEBUG { 04:38:37 DEBUG "status": "success", 04:38:37 DEBUG "data": { 04:38:37 DEBUG "resultType": "vector", 04:38:37 DEBUG "result": [ 04:38:37 DEBUG { 04:38:37 DEBUG "metric": { 04:38:37 DEBUG "pod": "ds-idrepo-0" 04:38:37 DEBUG }, 04:38:37 DEBUG "value": [ 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG [...] 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG "pod": "ds-idrepo-2" 04:38:37 DEBUG }, 04:38:37 DEBUG "value": [ 04:38:37 DEBUG 1684467517, 04:38:37 DEBUG "0" 04:38:37 DEBUG ] 04:38:37 DEBUG } 04:38:37 DEBUG ] 04:38:37 DEBUG } 04:38:37 DEBUG } 04:38:37 INFO [http_cmd]: http status code OK 04:38:37 DEBUG --- status code --- 04:38:37 DEBUG http status code is 200 (expected 200) 04:38:37 DEBUG --- http response --- 04:38:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:38:37 DEBUG ----- output ----- 04:38:37 DEBUG { 04:38:37 DEBUG "status": "success", 04:38:37 DEBUG "data": { 04:38:37 DEBUG "resultType": "vector", 04:38:37 DEBUG "result": [ 04:38:37 DEBUG { 04:38:37 DEBUG "metric": { 04:38:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:38:37 DEBUG }, 04:38:37 DEBUG "value": [ 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG [...] 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG 04:38:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:38:37 DEBUG }, 04:38:37 DEBUG "value": [ 04:38:37 DEBUG 1684467517, 04:38:37 DEBUG "0" 04:38:37 DEBUG ] 04:38:37 DEBUG } 04:38:37 DEBUG ] 04:38:37 DEBUG } 04:38:37 DEBUG } 04:38:50 INFO 04:38:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:38:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:38:50 INFO [loop_until]: OK (rc = 0) 04:38:50 DEBUG --- stdout --- 04:38:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7277m 9191Mi am-7849cf7bdb-jcbb8 6878m 9183Mi am-7849cf7bdb-zrhp7 7753m 9211Mi ds-cts-0 10542m 13851Mi ds-cts-1 10868m 13845Mi ds-cts-2 10378m 13847Mi ds-idrepo-0 4500m 13823Mi ds-idrepo-1 13m 13826Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3277Mi idm-78f4b47cb9-vh9b7 9m 3252Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1975m 854Mi 04:38:50 DEBUG --- stderr --- 04:38:50 DEBUG 04:38:55 INFO 04:38:55 INFO [loop_until]: kubectl --namespace=xlou top node 04:38:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:38:55 INFO [loop_until]: OK (rc = 0) 04:38:55 DEBUG --- stdout --- 04:38:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7924m 49% 11371Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 85m 0% 4889Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8310m 52% 11442Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13452m 84% 11851Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 162m 1% 3087Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 90m 0% 1329Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 94m 0% 5172Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 69m 0% 14802Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 10981m 69% 14684Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 67m 0% 14873Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4457m 28% 14807Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 10847m 68% 14666Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 10487m 65% 14716Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2103m 13% 2836Mi 4% 04:38:55 DEBUG --- stderr --- 04:38:55 DEBUG 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467576" 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG { 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "data": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "result": [ 04:39:36 DEBUG { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "pod": "ds-cts-2" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "pod": "ds-idrepo-2" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "0" 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG { 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "data": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "result": [ 04:39:36 DEBUG { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "instance": "10.142.0.17:9100" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "instance": "10.142.0.61:9100" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "0.0016999999999995907" 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG { 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "data": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "result": [ 04:39:36 DEBUG { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "pod": "end-user-ui-787cb4f6b4-56bl4" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "0" 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG { 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "data": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "result": [ 04:39:36 DEBUG { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "1030.0333333333333" 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467576" 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG { 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "data": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "result": [ 04:39:36 DEBUG { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "instance": "10.142.0.17:9100" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "instance": "10.142.0.61:9100" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "11757062.866666667" 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467576" 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG --- http response --- 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:36 DEBUG { 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:36 DEBUG { 04:39:36 DEBUG { 04:39:36 DEBUG "data": { 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG { 04:39:36 DEBUG "data": { 04:39:36 DEBUG "data": { 04:39:36 DEBUG "result": [ 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG { 04:39:36 DEBUG "data": { 04:39:36 DEBUG "result": [ 04:39:36 DEBUG "result": [ 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG { 04:39:36 DEBUG { 04:39:36 DEBUG "pod": "ds-idrepo-0" 04:39:36 DEBUG "result": [ 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG }, 04:39:36 DEBUG { 04:39:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:39:36 DEBUG "pod": "ds-idrepo-0" 04:39:36 DEBUG "value": [ 04:39:36 DEBUG "metric": { 04:39:36 DEBUG }, 04:39:36 DEBUG }, 04:39:36 DEBUG 04:39:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:39:36 DEBUG "value": [ 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG }, 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "pod": "ds-idrepo-2" 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG }, 04:39:36 DEBUG 04:39:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:39:36 DEBUG "pod": "ds-idrepo-2" 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG }, 04:39:36 DEBUG }, 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:39:36 DEBUG "value": [ 04:39:36 DEBUG "value": [ 04:39:36 DEBUG "0" 04:39:36 DEBUG }, 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG ] 04:39:36 DEBUG "value": [ 04:39:36 DEBUG "1031.0666666666666" 04:39:36 DEBUG "0" 04:39:36 DEBUG } 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG ] 04:39:36 DEBUG ] 04:39:36 DEBUG ] 04:39:36 DEBUG "1.9944748930237435" 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG ] 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG { 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "data": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "result": [ 04:39:36 DEBUG { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "341.96666666666664" 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467576" 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG { 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "data": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "result": [ 04:39:36 DEBUG { 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "14601773056" 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG { 04:39:36 DEBUG { 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "data": { 04:39:36 DEBUG "data": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "result": [ 04:39:36 DEBUG "result": [ 04:39:36 DEBUG { 04:39:36 DEBUG { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:39:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:39:36 DEBUG }, 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:39:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:39:36 DEBUG }, 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "346.23333333333335" 04:39:36 DEBUG "0.2" 04:39:36 DEBUG ] 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: http status code OK 04:39:36 DEBUG --- status code --- 04:39:36 DEBUG http status code is 200 (expected 200) 04:39:36 DEBUG --- http response --- 04:39:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:39:36 DEBUG ----- output ----- 04:39:36 DEBUG { 04:39:36 DEBUG "status": "success", 04:39:36 DEBUG "data": { 04:39:36 DEBUG "resultType": "vector", 04:39:36 DEBUG "result": [ 04:39:36 DEBUG { 04:39:36 DEBUG "metric": { 04:39:36 DEBUG "pod": "ds-cts-2" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG [...] 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG 04:39:36 DEBUG "pod": "ds-idrepo-2" 04:39:36 DEBUG }, 04:39:36 DEBUG "value": [ 04:39:36 DEBUG 1684467576, 04:39:36 DEBUG "0" 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG ] 04:39:36 DEBUG } 04:39:36 DEBUG } 04:39:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467576" 04:39:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467577" 04:39:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467577" 04:39:37 INFO [http_cmd]: http status code OK 04:39:37 DEBUG --- status code --- 04:39:37 DEBUG http status code is 200 (expected 200) 04:39:37 DEBUG --- http response --- 04:39:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:39:37 DEBUG ----- output ----- 04:39:37 DEBUG { 04:39:37 DEBUG "status": "success", 04:39:37 DEBUG "data": { 04:39:37 DEBUG "resultType": "vector", 04:39:37 DEBUG "result": [ 04:39:37 DEBUG { 04:39:37 DEBUG "metric": { 04:39:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG [...] 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG "pod": "end-user-ui-787cb4f6b4-56bl4" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 1684467576, 04:39:37 DEBUG "0.000014379019048498912" 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG } 04:39:37 INFO [http_cmd]: http status code OK 04:39:37 DEBUG --- status code --- 04:39:37 DEBUG http status code is 200 (expected 200) 04:39:37 DEBUG --- http response --- 04:39:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:39:37 INFO [http_cmd]: http status code OK 04:39:37 DEBUG ----- output ----- 04:39:37 DEBUG --- status code --- 04:39:37 DEBUG { 04:39:37 DEBUG http status code is 200 (expected 200) 04:39:37 DEBUG "status": "success", 04:39:37 DEBUG --- http response --- 04:39:37 DEBUG "data": { 04:39:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:37 DEBUG "resultType": "vector", 04:39:37 DEBUG ----- output ----- 04:39:37 DEBUG "result": [ 04:39:37 DEBUG { 04:39:37 DEBUG { 04:39:37 DEBUG "status": "success", 04:39:37 DEBUG "metric": { 04:39:37 DEBUG "data": { 04:39:37 DEBUG "pod": "ds-cts-2" 04:39:37 DEBUG "resultType": "vector", 04:39:37 DEBUG }, 04:39:37 DEBUG "result": [ 04:39:37 DEBUG "value": [ 04:39:37 DEBUG { 04:39:37 DEBUG 04:39:37 DEBUG "metric": { 04:39:37 DEBUG 04:39:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:39:37 DEBUG 04:39:37 DEBUG }, 04:39:37 DEBUG [...] 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG "pod": "ds-idrepo-2" 04:39:37 DEBUG [...] 04:39:37 DEBUG }, 04:39:37 DEBUG 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 04:39:37 DEBUG 1684467577, 04:39:37 DEBUG 04:39:37 DEBUG "0" 04:39:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:39:37 DEBUG ] 04:39:37 DEBUG }, 04:39:37 DEBUG } 04:39:37 DEBUG "value": [ 04:39:37 DEBUG ] 04:39:37 DEBUG 1684467577, 04:39:37 DEBUG } 04:39:37 DEBUG "0" 04:39:37 DEBUG } 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG } 04:39:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467577" 04:39:37 INFO [http_cmd]: http status code OK 04:39:37 DEBUG --- status code --- 04:39:37 DEBUG http status code is 200 (expected 200) 04:39:37 DEBUG --- http response --- 04:39:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:37 DEBUG ----- output ----- 04:39:37 DEBUG { 04:39:37 DEBUG "status": "success", 04:39:37 DEBUG "data": { 04:39:37 DEBUG "resultType": "vector", 04:39:37 DEBUG "result": [ 04:39:37 DEBUG { 04:39:37 DEBUG "metric": { 04:39:37 DEBUG "instance": "10.142.0.17:9100" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG [...] 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG "instance": "10.142.0.61:9100" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 1684467577, 04:39:37 DEBUG "11757715.466666667" 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG } 04:39:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467577" 04:39:37 INFO [http_cmd]: http status code OK 04:39:37 DEBUG --- status code --- 04:39:37 DEBUG http status code is 200 (expected 200) 04:39:37 DEBUG --- http response --- 04:39:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:37 DEBUG ----- output ----- 04:39:37 DEBUG { 04:39:37 DEBUG "status": "success", 04:39:37 DEBUG "data": { 04:39:37 DEBUG "resultType": "vector", 04:39:37 DEBUG "result": [ 04:39:37 DEBUG { 04:39:37 DEBUG "metric": { 04:39:37 DEBUG "instance": "10.142.0.17:9100" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG [...] 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG "instance": "10.142.0.61:9100" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 1684467577, 04:39:37 DEBUG "0.001333333333333009" 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG } 04:39:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467577" 04:39:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467577" 04:39:37 INFO [http_cmd]: http status code OK 04:39:37 DEBUG --- status code --- 04:39:37 DEBUG http status code is 200 (expected 200) 04:39:37 DEBUG --- http response --- 04:39:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:37 DEBUG ----- output ----- 04:39:37 DEBUG { 04:39:37 DEBUG "status": "success", 04:39:37 DEBUG "data": { 04:39:37 DEBUG "resultType": "vector", 04:39:37 DEBUG "result": [ 04:39:37 DEBUG { 04:39:37 DEBUG "metric": { 04:39:37 DEBUG "pod": "ds-idrepo-0" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG [...] 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG "pod": "ds-idrepo-2" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 1684467577, 04:39:37 DEBUG "0" 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG } 04:39:37 INFO [http_cmd]: http status code OK 04:39:37 DEBUG --- status code --- 04:39:37 DEBUG http status code is 200 (expected 200) 04:39:37 DEBUG --- http response --- 04:39:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:37 DEBUG ----- output ----- 04:39:37 DEBUG { 04:39:37 DEBUG "status": "success", 04:39:37 DEBUG "data": { 04:39:37 DEBUG "resultType": "vector", 04:39:37 DEBUG "result": [ 04:39:37 DEBUG { 04:39:37 DEBUG "metric": { 04:39:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG [...] 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG 04:39:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:39:37 DEBUG }, 04:39:37 DEBUG "value": [ 04:39:37 DEBUG 1684467577, 04:39:37 DEBUG "0" 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG ] 04:39:37 DEBUG } 04:39:37 DEBUG } 04:39:50 INFO 04:39:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:39:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:39:50 INFO [loop_until]: OK (rc = 0) 04:39:50 DEBUG --- stdout --- 04:39:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7555m 9207Mi am-7849cf7bdb-jcbb8 7061m 9185Mi am-7849cf7bdb-zrhp7 7376m 9213Mi ds-cts-0 10982m 13866Mi ds-cts-1 10833m 13898Mi ds-cts-2 11036m 13848Mi ds-idrepo-0 4197m 13823Mi ds-idrepo-1 16m 13826Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 8m 3253Mi lodemon-765d5f4748-dps7q 4m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1910m 854Mi 04:39:50 DEBUG --- stderr --- 04:39:50 DEBUG 04:39:55 INFO 04:39:55 INFO [loop_until]: kubectl --namespace=xlou top node 04:39:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:39:55 INFO [loop_until]: OK (rc = 0) 04:39:55 DEBUG --- stdout --- 04:39:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 8068m 50% 11376Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4894Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8468m 53% 11449Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13293m 83% 11817Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 163m 1% 3077Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 92m 0% 1332Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5173Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 67m 0% 14802Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 11079m 69% 14693Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 69m 0% 14877Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4469m 28% 14805Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 10924m 68% 14671Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 11002m 69% 14710Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1995m 12% 2839Mi 4% 04:39:55 DEBUG --- stderr --- 04:39:55 DEBUG 04:40:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467636" 04:40:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467636" 04:40:36 INFO [http_cmd]: http status code OK 04:40:36 DEBUG --- status code --- 04:40:36 DEBUG http status code is 200 (expected 200) 04:40:36 DEBUG --- http response --- 04:40:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:40:36 DEBUG ----- output ----- 04:40:36 DEBUG { 04:40:36 DEBUG "status": "success", 04:40:36 DEBUG "data": { 04:40:36 DEBUG "resultType": "vector", 04:40:36 DEBUG "result": [ 04:40:36 DEBUG { 04:40:36 DEBUG "metric": { 04:40:36 DEBUG "pod": "ds-cts-2" 04:40:36 DEBUG }, 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG [...] 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG "pod": "ds-idrepo-2" 04:40:36 DEBUG }, 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 1684467636, 04:40:36 DEBUG "0" 04:40:36 DEBUG ] 04:40:36 DEBUG } 04:40:36 DEBUG ] 04:40:36 DEBUG } 04:40:36 DEBUG } 04:40:36 INFO [http_cmd]: http status code OK 04:40:36 DEBUG --- status code --- 04:40:36 DEBUG http status code is 200 (expected 200) 04:40:36 DEBUG --- http response --- 04:40:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:36 DEBUG ----- output ----- 04:40:36 DEBUG { 04:40:36 DEBUG "status": "success", 04:40:36 DEBUG "data": { 04:40:36 DEBUG "resultType": "vector", 04:40:36 DEBUG "result": [ 04:40:36 DEBUG { 04:40:36 DEBUG "metric": { 04:40:36 DEBUG "instance": "10.142.0.17:9100" 04:40:36 DEBUG }, 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG [...] 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG "instance": "10.142.0.61:9100" 04:40:36 DEBUG }, 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 1684467636, 04:40:36 DEBUG "0.0014333333333316509" 04:40:36 DEBUG ] 04:40:36 DEBUG } 04:40:36 DEBUG ] 04:40:36 DEBUG } 04:40:36 DEBUG } 04:40:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467636" 04:40:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467636" 04:40:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467636" 04:40:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467636" 04:40:36 INFO [http_cmd]: http status code OK 04:40:36 DEBUG --- status code --- 04:40:36 DEBUG http status code is 200 (expected 200) 04:40:36 DEBUG --- http response --- 04:40:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:36 INFO [http_cmd]: http status code OK 04:40:36 DEBUG ----- output ----- 04:40:36 DEBUG --- status code --- 04:40:36 DEBUG { 04:40:36 DEBUG http status code is 200 (expected 200) 04:40:36 DEBUG "status": "success", 04:40:36 DEBUG --- http response --- 04:40:36 DEBUG "data": { 04:40:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:36 DEBUG "resultType": "vector", 04:40:36 DEBUG ----- output ----- 04:40:36 DEBUG "result": [ 04:40:36 DEBUG { 04:40:36 DEBUG { 04:40:36 DEBUG "status": "success", 04:40:36 DEBUG "metric": { 04:40:36 DEBUG "data": { 04:40:36 DEBUG "pod": "ds-idrepo-0" 04:40:36 DEBUG "resultType": "vector", 04:40:36 DEBUG }, 04:40:36 DEBUG "result": [ 04:40:36 DEBUG "value": [ 04:40:36 INFO [http_cmd]: http status code OK 04:40:36 DEBUG { 04:40:36 DEBUG 04:40:36 DEBUG --- status code --- 04:40:36 DEBUG "metric": { 04:40:36 DEBUG 04:40:36 DEBUG http status code is 200 (expected 200) 04:40:36 DEBUG "instance": "10.142.0.17:9100" 04:40:36 DEBUG 04:40:36 DEBUG --- http response --- 04:40:36 DEBUG }, 04:40:36 DEBUG [...] 04:40:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 04:40:36 DEBUG ----- output ----- 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG { 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG "status": "success", 04:40:36 DEBUG 04:40:36 DEBUG "pod": "ds-idrepo-2" 04:40:36 DEBUG "data": { 04:40:36 DEBUG [...] 04:40:36 DEBUG }, 04:40:36 DEBUG "resultType": "vector", 04:40:36 DEBUG 04:40:36 DEBUG "value": [ 04:40:36 DEBUG "result": [ 04:40:36 DEBUG 04:40:36 DEBUG 1684467636, 04:40:36 DEBUG { 04:40:36 DEBUG 04:40:36 DEBUG "0" 04:40:36 DEBUG "metric": { 04:40:36 DEBUG "instance": "10.142.0.61:9100" 04:40:36 DEBUG ] 04:40:36 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:40:36 DEBUG }, 04:40:36 DEBUG } 04:40:36 DEBUG }, 04:40:36 DEBUG "value": [ 04:40:36 DEBUG ] 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 1684467636, 04:40:36 DEBUG } 04:40:36 DEBUG 04:40:36 DEBUG "11706751.1" 04:40:36 DEBUG } 04:40:36 DEBUG 04:40:36 DEBUG ] 04:40:36 DEBUG 04:40:36 DEBUG } 04:40:36 DEBUG [...] 04:40:36 DEBUG ] 04:40:36 DEBUG 04:40:36 DEBUG } 04:40:36 DEBUG 04:40:36 DEBUG } 04:40:36 DEBUG 04:40:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:40:36 INFO [http_cmd]: http status code OK 04:40:36 DEBUG }, 04:40:36 DEBUG --- status code --- 04:40:36 DEBUG "value": [ 04:40:36 DEBUG http status code is 200 (expected 200) 04:40:36 DEBUG 1684467636, 04:40:36 DEBUG --- http response --- 04:40:36 DEBUG "0" 04:40:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:36 DEBUG ] 04:40:36 DEBUG ----- output ----- 04:40:36 DEBUG } 04:40:36 DEBUG { 04:40:36 DEBUG ] 04:40:36 DEBUG "status": "success", 04:40:36 DEBUG } 04:40:36 DEBUG "data": { 04:40:36 DEBUG } 04:40:36 DEBUG "resultType": "vector", 04:40:36 DEBUG "result": [ 04:40:36 DEBUG { 04:40:36 DEBUG "metric": { 04:40:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:40:36 DEBUG }, 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG [...] 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:40:36 DEBUG }, 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 1684467636, 04:40:36 DEBUG "1025.5333333333333" 04:40:36 DEBUG ] 04:40:36 DEBUG } 04:40:36 DEBUG ] 04:40:36 DEBUG } 04:40:36 DEBUG } 04:40:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467636" 04:40:36 INFO [http_cmd]: http status code OK 04:40:36 DEBUG --- status code --- 04:40:36 DEBUG http status code is 200 (expected 200) 04:40:36 DEBUG --- http response --- 04:40:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:36 DEBUG ----- output ----- 04:40:36 DEBUG { 04:40:36 DEBUG "status": "success", 04:40:36 DEBUG "data": { 04:40:36 DEBUG "resultType": "vector", 04:40:36 DEBUG "result": [ 04:40:36 DEBUG { 04:40:36 DEBUG "metric": { 04:40:36 DEBUG "pod": "ds-idrepo-0" 04:40:36 DEBUG }, 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG [...] 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG 04:40:36 DEBUG "pod": "ds-idrepo-2" 04:40:36 DEBUG }, 04:40:36 DEBUG "value": [ 04:40:36 DEBUG 1684467636, 04:40:36 DEBUG "0" 04:40:36 DEBUG ] 04:40:36 DEBUG } 04:40:36 DEBUG ] 04:40:36 DEBUG } 04:40:36 DEBUG } 04:40:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467636" 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467637" 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467637" 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG { 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "data": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "result": [ 04:40:37 DEBUG { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 1684467636, 04:40:37 DEBUG "341.76666666666665" 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467637" 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467637" 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG { 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG "status": "success", 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467637" 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:37 DEBUG "data": { 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG { 04:40:37 DEBUG "result": [ 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG { 04:40:37 DEBUG "data": { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:40:37 DEBUG "result": [ 04:40:37 DEBUG }, 04:40:37 DEBUG { 04:40:37 DEBUG "value": [ 04:40:37 DEBUG "metric": { 04:40:37 DEBUG 04:40:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:40:37 DEBUG 04:40:37 DEBUG }, 04:40:37 DEBUG 04:40:37 DEBUG "value": [ 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:40:37 DEBUG 04:40:37 DEBUG }, 04:40:37 DEBUG 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:40:37 DEBUG "1003.2666666666667" 04:40:37 DEBUG }, 04:40:37 DEBUG ] 04:40:37 DEBUG "value": [ 04:40:37 DEBUG } 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG ] 04:40:37 DEBUG "1.9471933247476574" 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG --- http response --- 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG { 04:40:37 DEBUG { 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:37 DEBUG "data": { 04:40:37 DEBUG "data": { 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG { 04:40:37 DEBUG "result": [ 04:40:37 DEBUG "result": [ 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG { 04:40:37 DEBUG { 04:40:37 DEBUG "data": { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "pod": "ds-cts-2" 04:40:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:40:37 DEBUG "result": [ 04:40:37 DEBUG }, 04:40:37 DEBUG }, 04:40:37 DEBUG { 04:40:37 DEBUG "value": [ 04:40:37 DEBUG "value": [ 04:40:37 DEBUG "metric": { 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG }, 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "value": [ 04:40:37 DEBUG [...] 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG "pod": "ds-idrepo-2" 04:40:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:40:37 DEBUG 04:40:37 DEBUG }, 04:40:37 DEBUG }, 04:40:37 DEBUG 04:40:37 DEBUG "value": [ 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:40:37 DEBUG "0" 04:40:37 DEBUG "335.6333333333333" 04:40:37 DEBUG }, 04:40:37 DEBUG ] 04:40:37 DEBUG ] 04:40:37 DEBUG "value": [ 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG ] 04:40:37 DEBUG ] 04:40:37 DEBUG "14602883072" 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467637" 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467637" 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG { 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "data": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "result": [ 04:40:37 DEBUG { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "0.2" 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG { 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "data": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "result": [ 04:40:37 DEBUG { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "pod": "ds-cts-2" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "pod": "ds-idrepo-2" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "0" 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467637" 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG { 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "data": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "result": [ 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467637" 04:40:37 DEBUG { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "pod": "ds-cts-2" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "0" 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG { 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "data": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "result": [ 04:40:37 DEBUG { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "pod": "ds-cts-2" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "0" 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467637" 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG { 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "data": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "result": [ 04:40:37 DEBUG { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "instance": "10.142.0.17:9100" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "instance": "10.142.0.61:9100" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "11709298.9" 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467637" 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG { 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "data": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "result": [ 04:40:37 DEBUG { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "instance": "10.142.0.17:9100" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "instance": "10.142.0.61:9100" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "0.0006666666666667116" 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467637" 04:40:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467637" 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG { 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "data": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "result": [ 04:40:37 DEBUG { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "pod": "ds-idrepo-0" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "pod": "ds-idrepo-2" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "0" 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:37 INFO [http_cmd]: http status code OK 04:40:37 DEBUG --- status code --- 04:40:37 DEBUG http status code is 200 (expected 200) 04:40:37 DEBUG --- http response --- 04:40:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:40:37 DEBUG ----- output ----- 04:40:37 DEBUG { 04:40:37 DEBUG "status": "success", 04:40:37 DEBUG "data": { 04:40:37 DEBUG "resultType": "vector", 04:40:37 DEBUG "result": [ 04:40:37 DEBUG { 04:40:37 DEBUG "metric": { 04:40:37 DEBUG "pod": "ds-cts-1" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG [...] 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG 04:40:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:40:37 DEBUG }, 04:40:37 DEBUG "value": [ 04:40:37 DEBUG 1684467637, 04:40:37 DEBUG "0" 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG ] 04:40:37 DEBUG } 04:40:37 DEBUG } 04:40:50 INFO 04:40:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:40:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:40:50 INFO [loop_until]: OK (rc = 0) 04:40:50 DEBUG --- stdout --- 04:40:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7214m 9207Mi am-7849cf7bdb-jcbb8 7135m 9186Mi am-7849cf7bdb-zrhp7 7702m 9215Mi ds-cts-0 10616m 13870Mi ds-cts-1 10859m 13886Mi ds-cts-2 10768m 13849Mi ds-idrepo-0 4543m 13822Mi ds-idrepo-1 12m 13826Mi ds-idrepo-2 15m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 7m 3259Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1979m 861Mi 04:40:50 DEBUG --- stderr --- 04:40:50 DEBUG 04:40:55 INFO 04:40:55 INFO [loop_until]: kubectl --namespace=xlou top node 04:40:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:40:55 INFO [loop_until]: OK (rc = 0) 04:40:55 DEBUG --- stdout --- 04:40:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7923m 49% 11373Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 76m 0% 4901Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8069m 50% 11450Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13352m 84% 11891Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 184m 1% 3076Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 86m 0% 1333Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 93m 0% 5170Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14804Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 10700m 67% 14703Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 63m 0% 14878Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4442m 27% 14808Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 10570m 66% 14714Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 10203m 64% 14714Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2012m 12% 2840Mi 4% 04:40:55 DEBUG --- stderr --- 04:40:55 DEBUG 04:41:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467696" 04:41:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467696" 04:41:36 INFO [http_cmd]: http status code OK 04:41:36 DEBUG --- status code --- 04:41:36 DEBUG http status code is 200 (expected 200) 04:41:36 DEBUG --- http response --- 04:41:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:41:36 DEBUG ----- output ----- 04:41:36 DEBUG { 04:41:36 DEBUG "status": "success", 04:41:36 DEBUG "data": { 04:41:36 DEBUG "resultType": "vector", 04:41:36 DEBUG "result": [ 04:41:36 DEBUG { 04:41:36 DEBUG "metric": { 04:41:36 DEBUG "pod": "ds-cts-2" 04:41:36 DEBUG }, 04:41:36 DEBUG "value": [ 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG [...] 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG "pod": "ds-idrepo-2" 04:41:36 DEBUG }, 04:41:36 DEBUG "value": [ 04:41:36 DEBUG 1684467696, 04:41:36 DEBUG "0" 04:41:36 DEBUG ] 04:41:36 DEBUG } 04:41:36 DEBUG ] 04:41:36 DEBUG } 04:41:36 DEBUG } 04:41:36 INFO [http_cmd]: http status code OK 04:41:36 DEBUG --- status code --- 04:41:36 DEBUG http status code is 200 (expected 200) 04:41:36 DEBUG --- http response --- 04:41:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:36 DEBUG ----- output ----- 04:41:36 DEBUG { 04:41:36 DEBUG "status": "success", 04:41:36 DEBUG "data": { 04:41:36 DEBUG "resultType": "vector", 04:41:36 DEBUG "result": [ 04:41:36 DEBUG { 04:41:36 DEBUG "metric": { 04:41:36 DEBUG "instance": "10.142.0.17:9100" 04:41:36 DEBUG }, 04:41:36 DEBUG "value": [ 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG [...] 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG "instance": "10.142.0.61:9100" 04:41:36 DEBUG }, 04:41:36 DEBUG "value": [ 04:41:36 DEBUG 1684467696, 04:41:36 DEBUG "0.0018000000000000683" 04:41:36 DEBUG ] 04:41:36 DEBUG } 04:41:36 DEBUG ] 04:41:36 DEBUG } 04:41:36 DEBUG } 04:41:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467696" 04:41:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467696" 04:41:36 INFO [http_cmd]: http status code OK 04:41:36 DEBUG --- status code --- 04:41:36 DEBUG http status code is 200 (expected 200) 04:41:36 DEBUG --- http response --- 04:41:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:36 DEBUG ----- output ----- 04:41:36 DEBUG { 04:41:36 DEBUG "status": "success", 04:41:36 DEBUG "data": { 04:41:36 DEBUG "resultType": "vector", 04:41:36 DEBUG "result": [ 04:41:36 DEBUG { 04:41:36 DEBUG "metric": { 04:41:36 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:41:36 DEBUG }, 04:41:36 DEBUG "value": [ 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG [...] 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG 04:41:36 DEBUG "pod": "am-7849cf7bdb-776xr" 04:41:36 DEBUG }, 04:41:36 DEBUG "value": [ 04:41:36 DEBUG 1684467696, 04:41:36 DEBUG "1020.0333333333333" 04:41:36 DEBUG ] 04:41:36 DEBUG } 04:41:36 DEBUG ] 04:41:36 DEBUG } 04:41:36 DEBUG } 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "ds-idrepo-0" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "ds-idrepo-2" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467696, 04:41:37 DEBUG "0" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467697" 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "instance": "10.142.0.17:9100" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "instance": "10.142.0.61:9100" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "11965103.833333334" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG { 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG "data": { 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "data": { 04:41:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG }, 04:41:37 DEBUG "result": [ 04:41:37 DEBUG "value": [ 04:41:37 DEBUG { 04:41:37 DEBUG 04:41:37 DEBUG "metric": { 04:41:37 DEBUG 04:41:37 DEBUG "pod": "ds-idrepo-0" 04:41:37 DEBUG 04:41:37 DEBUG }, 04:41:37 DEBUG [...] 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:41:37 DEBUG [...] 04:41:37 DEBUG }, 04:41:37 DEBUG 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG 04:41:37 DEBUG "482.5376610667101" 04:41:37 DEBUG "pod": "ds-idrepo-2" 04:41:37 DEBUG ] 04:41:37 DEBUG }, 04:41:37 DEBUG } 04:41:37 DEBUG "value": [ 04:41:37 DEBUG ] 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG } 04:41:37 DEBUG "0" 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "ds-cts-2" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "ds-idrepo-2" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "0" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:41:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:41:37 DEBUG }, 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:41:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:41:37 DEBUG }, 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG "value": [ 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467697" 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "1027.6" 04:41:37 DEBUG "340.3666666666667" 04:41:37 DEBUG ] 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "1.920038005114224" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467697" 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:41:37 DEBUG }, 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG "value": [ 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG "343.8" 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG ] 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:37 DEBUG } 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG ] 04:41:37 DEBUG { 04:41:37 DEBUG } 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG } 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "14573883392" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "0.2" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "ds-cts-2" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "ds-idrepo-2" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "0" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "0.004885642061737927" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "0" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467697" 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "instance": "10.142.0.17:9100" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "instance": "10.142.0.61:9100" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "11967569.233333332" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "0" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467697" 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467697" 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "instance": "10.142.0.17:9100" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "instance": "10.142.0.61:9100" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "0.0013333333333337195" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:37 INFO [http_cmd]: http status code OK 04:41:37 DEBUG --- status code --- 04:41:37 DEBUG http status code is 200 (expected 200) 04:41:37 DEBUG --- http response --- 04:41:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:37 DEBUG ----- output ----- 04:41:37 DEBUG { 04:41:37 DEBUG "status": "success", 04:41:37 DEBUG "data": { 04:41:37 DEBUG "resultType": "vector", 04:41:37 DEBUG "result": [ 04:41:37 DEBUG { 04:41:37 DEBUG "metric": { 04:41:37 DEBUG "pod": "ds-idrepo-0" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG [...] 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG 04:41:37 DEBUG "pod": "ds-idrepo-2" 04:41:37 DEBUG }, 04:41:37 DEBUG "value": [ 04:41:37 DEBUG 1684467697, 04:41:37 DEBUG "0" 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG ] 04:41:37 DEBUG } 04:41:37 DEBUG } 04:41:50 INFO 04:41:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:41:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:41:50 INFO [loop_until]: OK (rc = 0) 04:41:50 DEBUG --- stdout --- 04:41:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7259m 9209Mi am-7849cf7bdb-jcbb8 7605m 9188Mi am-7849cf7bdb-zrhp7 7597m 9259Mi ds-cts-0 10916m 13840Mi ds-cts-1 11351m 13886Mi ds-cts-2 11313m 13849Mi ds-idrepo-0 4549m 13822Mi ds-idrepo-1 12m 13826Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3277Mi idm-78f4b47cb9-vh9b7 10m 3259Mi lodemon-765d5f4748-dps7q 3m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 2133m 855Mi 04:41:50 DEBUG --- stderr --- 04:41:50 DEBUG 04:41:55 INFO 04:41:55 INFO [loop_until]: kubectl --namespace=xlou top node 04:41:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:41:55 INFO [loop_until]: OK (rc = 0) 04:41:55 DEBUG --- stdout --- 04:41:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 8447m 53% 11381Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 81m 0% 4896Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8206m 51% 11450Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 12882m 81% 11889Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 156m 0% 3068Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 87m 0% 1329Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5172Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14807Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 11409m 71% 14703Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 63m 0% 14879Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4731m 29% 14806Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 11169m 70% 14674Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 11219m 70% 14679Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2055m 12% 2841Mi 4% 04:41:55 DEBUG --- stderr --- 04:41:55 DEBUG 04:42:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467756" 04:42:36 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467756" 04:42:36 INFO [http_cmd]: http status code OK 04:42:36 DEBUG --- status code --- 04:42:36 DEBUG http status code is 200 (expected 200) 04:42:36 DEBUG --- http response --- 04:42:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:42:36 DEBUG ----- output ----- 04:42:36 DEBUG { 04:42:36 DEBUG "status": "success", 04:42:36 DEBUG "data": { 04:42:36 DEBUG "resultType": "vector", 04:42:36 DEBUG "result": [ 04:42:36 DEBUG { 04:42:36 DEBUG "metric": { 04:42:36 DEBUG "pod": "ds-cts-2" 04:42:36 DEBUG }, 04:42:36 DEBUG "value": [ 04:42:36 DEBUG 04:42:36 DEBUG 04:42:36 DEBUG 04:42:36 DEBUG [...] 04:42:36 DEBUG 04:42:36 DEBUG 04:42:36 DEBUG 04:42:36 DEBUG "pod": "ds-idrepo-2" 04:42:36 DEBUG }, 04:42:36 DEBUG "value": [ 04:42:36 DEBUG 1684467756, 04:42:36 DEBUG "0" 04:42:36 DEBUG ] 04:42:36 DEBUG } 04:42:36 DEBUG ] 04:42:36 DEBUG } 04:42:36 DEBUG } 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "instance": "10.142.0.17:9100" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "instance": "10.142.0.61:9100" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467756, 04:42:37 DEBUG "0.0015666666666656207" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "ds-idrepo-0" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "ds-idrepo-2" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "0" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "1018.3" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "ds-cts-2" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "ds-idrepo-2" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "0" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "instance": "10.142.0.17:9100" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "instance": "10.142.0.61:9100" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "11750956.066666666" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "ds-idrepo-0" 04:42:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:42:37 DEBUG }, 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG "pod": "ds-idrepo-2" 04:42:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG }, 04:42:37 DEBUG }, 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG "value": [ 04:42:37 DEBUG "value": [ 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG "0" 04:42:37 DEBUG "1017.9" 04:42:37 DEBUG { 04:42:37 DEBUG ] 04:42:37 DEBUG ] 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 DEBUG "data": { 04:42:37 DEBUG ] 04:42:37 DEBUG ] 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 DEBUG "result": [ 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "0" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467757" 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "337.73333333333335" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "14482817024" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "2.0343471583038526" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "340.6" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG { 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:42:37 DEBUG "data": { 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG { 04:42:37 DEBUG "result": [ 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG { 04:42:37 DEBUG "data": { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:42:37 DEBUG "result": [ 04:42:37 DEBUG }, 04:42:37 DEBUG { 04:42:37 DEBUG "value": [ 04:42:37 DEBUG "metric": { 04:42:37 DEBUG 04:42:37 DEBUG "pod": "ds-cts-2" 04:42:37 DEBUG 04:42:37 DEBUG }, 04:42:37 DEBUG 04:42:37 DEBUG "value": [ 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:42:37 DEBUG 04:42:37 DEBUG }, 04:42:37 DEBUG 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "pod": "ds-idrepo-2" 04:42:37 DEBUG "0.2" 04:42:37 DEBUG }, 04:42:37 DEBUG ] 04:42:37 DEBUG "value": [ 04:42:37 DEBUG } 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG ] 04:42:37 DEBUG "0" 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "0" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "instance": "10.142.0.17:9100" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "instance": "10.142.0.61:9100" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "11753551.566666666" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "ds-cts-2" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "658.3451039647866" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "ds-cts-2" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "58.69986532207732" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467757" 04:42:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467757" 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "instance": "10.142.0.17:9100" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "instance": "10.142.0.61:9100" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "0.00033333333333350387" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:37 INFO [http_cmd]: http status code OK 04:42:37 DEBUG --- status code --- 04:42:37 DEBUG http status code is 200 (expected 200) 04:42:37 DEBUG --- http response --- 04:42:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:37 DEBUG ----- output ----- 04:42:37 DEBUG { 04:42:37 DEBUG "status": "success", 04:42:37 DEBUG "data": { 04:42:37 DEBUG "resultType": "vector", 04:42:37 DEBUG "result": [ 04:42:37 DEBUG { 04:42:37 DEBUG "metric": { 04:42:37 DEBUG "pod": "ds-idrepo-0" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG [...] 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG 04:42:37 DEBUG "pod": "ds-idrepo-2" 04:42:37 DEBUG }, 04:42:37 DEBUG "value": [ 04:42:37 DEBUG 1684467757, 04:42:37 DEBUG "0" 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG ] 04:42:37 DEBUG } 04:42:37 DEBUG } 04:42:50 INFO 04:42:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:42:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:42:50 INFO [loop_until]: OK (rc = 0) 04:42:50 DEBUG --- stdout --- 04:42:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 6651m 9208Mi am-7849cf7bdb-jcbb8 6601m 9188Mi am-7849cf7bdb-zrhp7 7376m 9216Mi ds-cts-0 10218m 13875Mi ds-cts-1 10248m 13887Mi ds-cts-2 10255m 13847Mi ds-idrepo-0 4261m 13823Mi ds-idrepo-1 13m 13825Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 5m 3278Mi idm-78f4b47cb9-vh9b7 8m 3253Mi lodemon-765d5f4748-dps7q 4m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1938m 857Mi 04:42:50 DEBUG --- stderr --- 04:42:50 DEBUG 04:42:55 INFO 04:42:55 INFO [loop_until]: kubectl --namespace=xlou top node 04:42:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:42:55 INFO [loop_until]: OK (rc = 0) 04:42:55 DEBUG --- stdout --- 04:42:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7433m 46% 11378Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 83m 0% 4900Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 7615m 47% 11452Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13011m 81% 11845Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 158m 0% 3080Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 85m 0% 1330Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5170Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14805Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 9870m 62% 14703Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 64m 0% 14880Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4537m 28% 14807Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 10505m 66% 14665Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 10075m 63% 14715Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2015m 12% 2842Mi 4% 04:42:55 DEBUG --- stderr --- 04:42:55 DEBUG 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "pod": "ds-cts-2" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "ds-idrepo-2" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "0" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "instance": "10.142.0.17:9100" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "instance": "10.142.0.61:9100" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "0.0017666666666665757" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "pod": "ds-idrepo-0" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "ds-idrepo-2" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "0" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG { 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG "data": { 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "data": { 04:43:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG }, 04:43:37 DEBUG "result": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG { 04:43:37 DEBUG 04:43:37 DEBUG "metric": { 04:43:37 DEBUG 04:43:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:43:37 DEBUG 04:43:37 DEBUG }, 04:43:37 DEBUG [...] 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:43:37 DEBUG [...] 04:43:37 DEBUG }, 04:43:37 DEBUG 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG 04:43:37 DEBUG "1018.7333333333333" 04:43:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:43:37 DEBUG ] 04:43:37 DEBUG }, 04:43:37 DEBUG } 04:43:37 DEBUG "value": [ 04:43:37 DEBUG ] 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG } 04:43:37 DEBUG "1019.0666666666666" 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "data": { 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG "result": [ 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "instance": "10.142.0.17:9100" 04:43:37 DEBUG "pod": "ds-cts-2" 04:43:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG [...] 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "instance": "10.142.0.61:9100" 04:43:37 DEBUG "pod": "ds-idrepo-2" 04:43:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "11818741" 04:43:37 DEBUG "0" 04:43:37 DEBUG "105.39326082808624" 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467817" 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "14600155136" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "pod": "ds-idrepo-0" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "ds-idrepo-2" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "0" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "339.3" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:43:37 DEBUG "pod": "ds-cts-2" 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:43:37 DEBUG "pod": "ds-idrepo-2" 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "340.06666666666666" 04:43:37 DEBUG "0" 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "1.9369651182409982" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:43:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:43:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "0.2" 04:43:37 DEBUG "0" 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:43:37 DEBUG "instance": "10.142.0.17:9100" 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:43:37 DEBUG "instance": "10.142.0.61:9100" 04:43:37 DEBUG }, 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "3.2226478531152383" 04:43:37 DEBUG "11821385.133333333" 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "0" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467817" 04:43:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467817" 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "instance": "10.142.0.17:9100" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "instance": "10.142.0.61:9100" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "0.0006666666666667116" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:37 INFO [http_cmd]: http status code OK 04:43:37 DEBUG --- status code --- 04:43:37 DEBUG http status code is 200 (expected 200) 04:43:37 DEBUG --- http response --- 04:43:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:37 DEBUG ----- output ----- 04:43:37 DEBUG { 04:43:37 DEBUG "status": "success", 04:43:37 DEBUG "data": { 04:43:37 DEBUG "resultType": "vector", 04:43:37 DEBUG "result": [ 04:43:37 DEBUG { 04:43:37 DEBUG "metric": { 04:43:37 DEBUG "pod": "ds-idrepo-0" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG [...] 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG 04:43:37 DEBUG "pod": "ds-idrepo-2" 04:43:37 DEBUG }, 04:43:37 DEBUG "value": [ 04:43:37 DEBUG 1684467817, 04:43:37 DEBUG "0" 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG ] 04:43:37 DEBUG } 04:43:37 DEBUG } 04:43:50 INFO 04:43:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:43:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:43:50 INFO [loop_until]: OK (rc = 0) 04:43:50 DEBUG --- stdout --- 04:43:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7286m 9208Mi am-7849cf7bdb-jcbb8 7403m 9190Mi am-7849cf7bdb-zrhp7 7501m 9214Mi ds-cts-0 11041m 13878Mi ds-cts-1 11457m 13888Mi ds-cts-2 10721m 13836Mi ds-idrepo-0 4902m 13823Mi ds-idrepo-1 13m 13826Mi ds-idrepo-2 15m 13827Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 5m 3278Mi idm-78f4b47cb9-vh9b7 9m 3259Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1909m 857Mi 04:43:50 DEBUG --- stderr --- 04:43:50 DEBUG 04:43:55 INFO 04:43:55 INFO [loop_until]: kubectl --namespace=xlou top node 04:43:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:43:55 INFO [loop_until]: OK (rc = 0) 04:43:55 DEBUG --- stdout --- 04:43:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 8220m 51% 11385Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 83m 0% 4899Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8155m 51% 11450Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13250m 83% 11819Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 158m 0% 3084Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 89m 0% 1327Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5168Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 69m 0% 14804Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 11068m 69% 14699Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 68m 0% 14876Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 4923m 30% 14801Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 11091m 69% 14663Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 11015m 69% 14712Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2012m 12% 2841Mi 4% 04:43:55 DEBUG --- stderr --- 04:43:55 DEBUG 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "pod": "ds-cts-2" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "ds-idrepo-2" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "0" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "instance": "10.142.0.17:9100" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "instance": "10.142.0.61:9100" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "0.0020333333333335682" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:44:37 DEBUG "pod": "ds-idrepo-0" 04:44:37 DEBUG }, 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:44:37 DEBUG "pod": "ds-idrepo-2" 04:44:37 DEBUG }, 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "0" 04:44:37 DEBUG "1036.3333333333333" 04:44:37 DEBUG ] 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467877" 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG ----- output ----- 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:44:37 DEBUG { 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG { 04:44:37 DEBUG "data": { 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:37 DEBUG "data": { 04:44:37 DEBUG "result": [ 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG { 04:44:37 DEBUG { 04:44:37 DEBUG "result": [ 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG { 04:44:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:44:37 DEBUG "data": { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG }, 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "pod": "ds-cts-1" 04:44:37 DEBUG "value": [ 04:44:37 DEBUG "result": [ 04:44:37 DEBUG }, 04:44:37 DEBUG 04:44:37 DEBUG { 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG "metric": { 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "ds-idrepo-0" 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG }, 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "value": [ 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG }, 04:44:37 DEBUG [...] 04:44:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG }, 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG 04:44:37 DEBUG "value": [ 04:44:37 DEBUG "1035.6" 04:44:37 DEBUG 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG ] 04:44:37 DEBUG "pod": "ds-idrepo-2" 04:44:37 DEBUG "52633.65508173206" 04:44:37 DEBUG } 04:44:37 DEBUG }, 04:44:37 DEBUG ] 04:44:37 DEBUG ] 04:44:37 DEBUG "value": [ 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG "0" 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG { 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG "data": { 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "data": { 04:44:37 DEBUG "pod": "ds-cts-2" 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG }, 04:44:37 DEBUG "result": [ 04:44:37 DEBUG "value": [ 04:44:37 DEBUG { 04:44:37 DEBUG 04:44:37 DEBUG "metric": { 04:44:37 DEBUG 04:44:37 DEBUG "instance": "10.142.0.17:9100" 04:44:37 DEBUG 04:44:37 DEBUG }, 04:44:37 DEBUG [...] 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "ds-idrepo-2" 04:44:37 DEBUG [...] 04:44:37 DEBUG }, 04:44:37 DEBUG 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG 04:44:37 DEBUG "0" 04:44:37 DEBUG "instance": "10.142.0.61:9100" 04:44:37 DEBUG ] 04:44:37 DEBUG }, 04:44:37 DEBUG } 04:44:37 DEBUG "value": [ 04:44:37 DEBUG ] 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG } 04:44:37 DEBUG "11797033" 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "14601687040" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "347.43333333333334" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG { 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG "data": { 04:44:37 DEBUG { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG "data": { 04:44:37 DEBUG { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "result": [ 04:44:37 DEBUG "pod": "ds-cts-2" 04:44:37 DEBUG { 04:44:37 DEBUG }, 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "value": [ 04:44:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:44:37 DEBUG 04:44:37 DEBUG }, 04:44:37 DEBUG 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "ds-idrepo-2" 04:44:37 DEBUG 04:44:37 DEBUG }, 04:44:37 DEBUG 04:44:37 DEBUG "value": [ 04:44:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG }, 04:44:37 DEBUG "0" 04:44:37 DEBUG "value": [ 04:44:37 DEBUG ] 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG } 04:44:37 DEBUG "344.4" 04:44:37 DEBUG ] 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "instance": "10.142.0.17:9100" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "instance": "10.142.0.61:9100" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "11797630.633333333" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "1.9080061146759666" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:44:37 DEBUG "pod": "ds-cts-1" 04:44:37 DEBUG }, 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:44:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:44:37 DEBUG }, 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "0.2" 04:44:37 DEBUG "0" 04:44:37 DEBUG ] 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "pod": "ds-cts-1" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "7.187729503601081" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "pod": "ds-cts-1" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "0" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467877" 04:44:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467877" 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "instance": "10.142.0.17:9100" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "instance": "10.142.0.61:9100" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "0.0013333333333331273" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:37 INFO [http_cmd]: http status code OK 04:44:37 DEBUG --- status code --- 04:44:37 DEBUG http status code is 200 (expected 200) 04:44:37 DEBUG --- http response --- 04:44:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:37 DEBUG ----- output ----- 04:44:37 DEBUG { 04:44:37 DEBUG "status": "success", 04:44:37 DEBUG "data": { 04:44:37 DEBUG "resultType": "vector", 04:44:37 DEBUG "result": [ 04:44:37 DEBUG { 04:44:37 DEBUG "metric": { 04:44:37 DEBUG "pod": "ds-idrepo-0" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG [...] 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG 04:44:37 DEBUG "pod": "ds-idrepo-2" 04:44:37 DEBUG }, 04:44:37 DEBUG "value": [ 04:44:37 DEBUG 1684467877, 04:44:37 DEBUG "0" 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG ] 04:44:37 DEBUG } 04:44:37 DEBUG } 04:44:50 INFO 04:44:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:44:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:44:50 INFO [loop_until]: OK (rc = 0) 04:44:50 DEBUG --- stdout --- 04:44:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 7535m 9210Mi am-7849cf7bdb-jcbb8 7018m 9192Mi am-7849cf7bdb-zrhp7 7799m 9225Mi ds-cts-0 11221m 13831Mi ds-cts-1 11458m 13889Mi ds-cts-2 11521m 13799Mi ds-idrepo-0 4863m 13822Mi ds-idrepo-1 13m 13826Mi ds-idrepo-2 15m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 6m 3277Mi idm-78f4b47cb9-vh9b7 8m 3261Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 2122m 858Mi 04:44:50 DEBUG --- stderr --- 04:44:50 DEBUG 04:44:55 INFO 04:44:55 INFO [loop_until]: kubectl --namespace=xlou top node 04:44:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:44:56 INFO [loop_until]: OK (rc = 0) 04:44:56 DEBUG --- stdout --- 04:44:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 7763m 48% 11384Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 74m 0% 4900Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 8304m 52% 11453Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 13393m 84% 11826Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 162m 1% 3061Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 83m 0% 1335Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5172Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14806Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 11447m 72% 14702Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 66m 0% 14879Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 5080m 31% 14804Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 11651m 73% 14668Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 11440m 71% 14714Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 2146m 13% 2841Mi 4% 04:44:56 DEBUG --- stderr --- 04:44:56 DEBUG 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "pod": "ds-cts-2" 04:45:37 DEBUG "instance": "10.142.0.17:9100" 04:45:37 DEBUG }, 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "ds-idrepo-2" 04:45:37 DEBUG "instance": "10.142.0.61:9100" 04:45:37 DEBUG }, 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0" 04:45:37 DEBUG "0.0004000000000009626" 04:45:37 DEBUG ] 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:45:37 DEBUG "pod": "ds-idrepo-0" 04:45:37 DEBUG }, 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:45:37 DEBUG "pod": "ds-idrepo-2" 04:45:37 DEBUG }, 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0" 04:45:37 DEBUG "0" 04:45:37 DEBUG ] 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467937" 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0" 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG { 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG "data": { 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "data": { 04:45:37 DEBUG "pod": "ds-idrepo-0" 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG }, 04:45:37 DEBUG "result": [ 04:45:37 DEBUG "value": [ 04:45:37 DEBUG { 04:45:37 DEBUG 04:45:37 DEBUG "metric": { 04:45:37 DEBUG 04:45:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:45:37 DEBUG 04:45:37 DEBUG }, 04:45:37 DEBUG [...] 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "ds-idrepo-2" 04:45:37 DEBUG [...] 04:45:37 DEBUG }, 04:45:37 DEBUG 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG 04:45:37 DEBUG "0" 04:45:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:45:37 DEBUG ] 04:45:37 DEBUG }, 04:45:37 DEBUG } 04:45:37 DEBUG "value": [ 04:45:37 DEBUG ] 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG } 04:45:37 DEBUG "275780074.7320709" 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:37 DEBUG { 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG { 04:45:37 DEBUG "data": { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "result": [ 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG { 04:45:37 DEBUG "result": [ 04:45:37 DEBUG "metric": { 04:45:37 DEBUG { 04:45:37 DEBUG "pod": "ds-cts-2" 04:45:37 DEBUG "metric": { 04:45:37 DEBUG }, 04:45:37 DEBUG "instance": "10.142.0.17:9100" 04:45:37 DEBUG "value": [ 04:45:37 DEBUG }, 04:45:37 DEBUG 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "ds-idrepo-2" 04:45:37 DEBUG 04:45:37 DEBUG }, 04:45:37 DEBUG "instance": "10.142.0.61:9100" 04:45:37 DEBUG "value": [ 04:45:37 DEBUG }, 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG "0" 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG ] 04:45:37 DEBUG "4680.433333333333" 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467937" 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "14601428992" 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "instance": "10.142.0.17:9100" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "instance": "10.142.0.61:9100" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "7645.266666666666" 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0" 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:45:37 DEBUG "pod": "ds-cts-2" 04:45:37 DEBUG }, 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:45:37 DEBUG "pod": "ds-idrepo-2" 04:45:37 DEBUG }, 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0" 04:45:37 DEBUG "0" 04:45:37 DEBUG ] 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "1.5459120165674627" 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:45:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:45:37 DEBUG }, 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:45:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:45:37 DEBUG }, 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0.2" 04:45:37 DEBUG "0" 04:45:37 DEBUG ] 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0.4281578422465284" 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0" 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "instance": "10.142.0.17:9100" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "instance": "10.142.0.61:9100" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0" 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467937" 04:45:37 INFO [http_cmd]: http status code OK 04:45:37 DEBUG --- status code --- 04:45:37 DEBUG http status code is 200 (expected 200) 04:45:37 DEBUG --- http response --- 04:45:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:37 DEBUG ----- output ----- 04:45:37 DEBUG { 04:45:37 DEBUG "status": "success", 04:45:37 DEBUG "data": { 04:45:37 DEBUG "resultType": "vector", 04:45:37 DEBUG "result": [ 04:45:37 DEBUG { 04:45:37 DEBUG "metric": { 04:45:37 DEBUG "pod": "ds-idrepo-0" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG [...] 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG 04:45:37 DEBUG "pod": "ds-idrepo-2" 04:45:37 DEBUG }, 04:45:37 DEBUG "value": [ 04:45:37 DEBUG 1684467937, 04:45:37 DEBUG "0" 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG ] 04:45:37 DEBUG } 04:45:37 DEBUG } 04:45:50 INFO 04:45:50 INFO [loop_until]: kubectl --namespace=xlou top pods 04:45:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:45:51 INFO [loop_until]: OK (rc = 0) 04:45:51 DEBUG --- stdout --- 04:45:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 4Mi am-7849cf7bdb-776xr 525m 9209Mi am-7849cf7bdb-jcbb8 520m 9190Mi am-7849cf7bdb-zrhp7 522m 9266Mi ds-cts-0 2150m 13876Mi ds-cts-1 2138m 13860Mi ds-cts-2 2375m 13823Mi ds-idrepo-0 384m 13822Mi ds-idrepo-1 14m 13826Mi ds-idrepo-2 14m 13826Mi end-user-ui-787cb4f6b4-56bl4 1m 4Mi idm-78f4b47cb9-bpwjf 5m 3277Mi idm-78f4b47cb9-vh9b7 8m 3263Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 2m 105Mi 04:45:51 DEBUG --- stderr --- 04:45:51 DEBUG 04:45:56 INFO 04:45:56 INFO [loop_until]: kubectl --namespace=xlou top node 04:45:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:45:56 INFO [loop_until]: OK (rc = 0) 04:45:56 DEBUG --- stdout --- 04:45:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 626m 3% 11387Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 93m 0% 4902Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 651m 4% 11450Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 631m 3% 11859Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 161m 1% 3079Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 87m 0% 1335Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 93m 0% 5172Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14805Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 2335m 14% 14679Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 67m 0% 14879Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 439m 2% 14804Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 2262m 14% 14650Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2182m 13% 14723Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 74m 0% 2098Mi 3% 04:45:56 DEBUG --- stderr --- 04:45:56 DEBUG 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "instance": "10.142.0.17:9100" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "instance": "10.142.0.61:9100" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG }, 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG "value": [ 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG "0.0006333333333335152" 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:46:37 DEBUG ] 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG } 04:46:37 DEBUG { 04:46:37 DEBUG ] 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG } 04:46:37 DEBUG "data": { 04:46:37 DEBUG } 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "ds-cts-2" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "ds-idrepo-2" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684467997" 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "ds-idrepo-0" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "ds-idrepo-2" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "14505254912" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "ds-idrepo-0" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "ds-idrepo-2" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG "data": { 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG "result": [ 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG { 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:37 DEBUG "metric": { 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG "pod": "ds-cts-2" 04:46:37 DEBUG { 04:46:37 DEBUG }, 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "value": [ 04:46:37 DEBUG "data": { 04:46:37 DEBUG 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG 04:46:37 DEBUG "result": [ 04:46:37 DEBUG 04:46:37 DEBUG { 04:46:37 DEBUG [...] 04:46:37 DEBUG "metric": { 04:46:37 DEBUG 04:46:37 DEBUG "instance": "10.142.0.17:9100" 04:46:37 DEBUG 04:46:37 DEBUG }, 04:46:37 DEBUG 04:46:37 DEBUG "value": [ 04:46:37 DEBUG "pod": "ds-idrepo-2" 04:46:37 DEBUG 04:46:37 DEBUG }, 04:46:37 DEBUG 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG [...] 04:46:37 DEBUG "0" 04:46:37 DEBUG 04:46:37 DEBUG ] 04:46:37 DEBUG 04:46:37 DEBUG } 04:46:37 DEBUG 04:46:37 DEBUG ] 04:46:37 DEBUG "instance": "10.142.0.61:9100" 04:46:37 DEBUG } 04:46:37 DEBUG }, 04:46:37 DEBUG } 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "4026.2333333333336" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "instance": "10.142.0.17:9100" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "instance": "10.142.0.61:9100" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "6631.400000000001" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "ds-cts-2" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "ds-idrepo-2" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0.028935423022645824" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG { 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:46:37 DEBUG "data": { 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG { 04:46:37 DEBUG "result": [ 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG { 04:46:37 DEBUG "data": { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:46:37 DEBUG "result": [ 04:46:37 DEBUG }, 04:46:37 DEBUG { 04:46:37 DEBUG "value": [ 04:46:37 DEBUG "metric": { 04:46:37 DEBUG 04:46:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:46:37 DEBUG 04:46:37 DEBUG }, 04:46:37 DEBUG 04:46:37 DEBUG "value": [ 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:46:37 DEBUG 04:46:37 DEBUG }, 04:46:37 DEBUG 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:46:37 DEBUG "0.2" 04:46:37 DEBUG }, 04:46:37 DEBUG ] 04:46:37 DEBUG "value": [ 04:46:37 DEBUG } 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG ] 04:46:37 DEBUG "0" 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0.3873750696868995" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "instance": "10.142.0.17:9100" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "instance": "10.142.0.61:9100" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684467997" 04:46:37 INFO [http_cmd]: http status code OK 04:46:37 DEBUG --- status code --- 04:46:37 DEBUG http status code is 200 (expected 200) 04:46:37 DEBUG --- http response --- 04:46:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:37 DEBUG ----- output ----- 04:46:37 DEBUG { 04:46:37 DEBUG "status": "success", 04:46:37 DEBUG "data": { 04:46:37 DEBUG "resultType": "vector", 04:46:37 DEBUG "result": [ 04:46:37 DEBUG { 04:46:37 DEBUG "metric": { 04:46:37 DEBUG "pod": "ds-idrepo-0" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG [...] 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG 04:46:37 DEBUG "pod": "ds-idrepo-2" 04:46:37 DEBUG }, 04:46:37 DEBUG "value": [ 04:46:37 DEBUG 1684467997, 04:46:37 DEBUG "0" 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG ] 04:46:37 DEBUG } 04:46:37 DEBUG } 04:46:51 INFO 04:46:51 INFO [loop_until]: kubectl --namespace=xlou top pods 04:46:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:46:51 INFO [loop_until]: OK (rc = 0) 04:46:51 DEBUG --- stdout --- 04:46:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 5Mi am-7849cf7bdb-776xr 509m 9131Mi am-7849cf7bdb-jcbb8 516m 9111Mi am-7849cf7bdb-zrhp7 524m 9148Mi ds-cts-0 2768m 13688Mi ds-cts-1 2836m 13686Mi ds-cts-2 2386m 13800Mi ds-idrepo-0 1011m 13776Mi ds-idrepo-1 294m 13817Mi ds-idrepo-2 350m 13785Mi end-user-ui-787cb4f6b4-56bl4 1m 5Mi idm-78f4b47cb9-bpwjf 8m 3277Mi idm-78f4b47cb9-vh9b7 8m 3265Mi lodemon-765d5f4748-dps7q 2m 67Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1050m 653Mi 04:46:51 DEBUG --- stderr --- 04:46:51 DEBUG 04:46:56 INFO 04:46:56 INFO [loop_until]: kubectl --namespace=xlou top node 04:46:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:46:56 INFO [loop_until]: OK (rc = 0) 04:46:56 DEBUG --- stdout --- 04:46:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 600m 3% 11323Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 83m 0% 4900Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 647m 4% 11396Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 622m 3% 11756Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 161m 1% 3035Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 89m 0% 1334Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 104m 0% 5178Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 447m 2% 14770Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 2476m 15% 14497Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 243m 1% 14835Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 1036m 6% 14763Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 2664m 16% 14643Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2410m 15% 14556Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 1083m 6% 2645Mi 4% 04:46:56 DEBUG --- stderr --- 04:46:56 DEBUG 127.0.0.1 - - [19/May/2023 04:47:10] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:47:16] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:47:23] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:47:29] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:47:36] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "instance": "10.142.0.17:9100" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "instance": "10.142.0.61:9100" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0.0013333333333340155" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "ds-cts-2" 04:47:37 DEBUG "pod": "ds-idrepo-0" 04:47:37 DEBUG }, 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "ds-idrepo-2" 04:47:37 DEBUG "pod": "ds-idrepo-2" 04:47:37 DEBUG }, 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468057" 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "60299.19018912835" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG "14421450752" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG ] 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG } 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:37 DEBUG ] 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG } 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:37 DEBUG { 04:47:37 DEBUG } 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG { 04:47:37 DEBUG "data": { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "result": [ 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG { 04:47:37 DEBUG "result": [ 04:47:37 DEBUG "metric": { 04:47:37 DEBUG { 04:47:37 DEBUG "pod": "ds-idrepo-0" 04:47:37 DEBUG "metric": { 04:47:37 DEBUG }, 04:47:37 DEBUG "instance": "10.142.0.17:9100" 04:47:37 DEBUG "value": [ 04:47:37 DEBUG }, 04:47:37 DEBUG 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG 04:47:37 DEBUG "pod": "ds-idrepo-2" 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG 04:47:37 DEBUG }, 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG "instance": "10.142.0.61:9100" 04:47:37 DEBUG "value": [ 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG }, 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:37 DEBUG "value": [ 04:47:37 DEBUG "0" 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG ] 04:47:37 DEBUG { 04:47:37 DEBUG "5279.166666666667" 04:47:37 DEBUG } 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG ] 04:47:37 DEBUG ] 04:47:37 DEBUG "data": { 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG "result": [ 04:47:37 DEBUG } 04:47:37 DEBUG { 04:47:37 DEBUG } 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:47:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:47:37 DEBUG }, 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:47:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:47:37 DEBUG }, 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "ds-cts-2" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "ds-idrepo-2" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468057" 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "instance": "10.142.0.17:9100" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "instance": "10.142.0.61:9100" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "7980.866666666667" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "ds-cts-2" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "ds-idrepo-2" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG "data": { 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG "result": [ 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG { 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:37 DEBUG "metric": { 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:47:37 DEBUG { 04:47:37 DEBUG }, 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "value": [ 04:47:37 DEBUG "data": { 04:47:37 DEBUG 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG 04:47:37 DEBUG "result": [ 04:47:37 DEBUG 04:47:37 DEBUG { 04:47:37 DEBUG [...] 04:47:37 DEBUG "metric": { 04:47:37 DEBUG 04:47:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:47:37 DEBUG 04:47:37 DEBUG }, 04:47:37 DEBUG 04:47:37 DEBUG "value": [ 04:47:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:47:37 DEBUG 04:47:37 DEBUG }, 04:47:37 DEBUG 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG [...] 04:47:37 DEBUG "0.009387214819797844" 04:47:37 DEBUG 04:47:37 DEBUG ] 04:47:37 DEBUG 04:47:37 DEBUG } 04:47:37 DEBUG 04:47:37 DEBUG ] 04:47:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:47:37 DEBUG } 04:47:37 DEBUG }, 04:47:37 DEBUG } 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0.9728352415421594" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0.2" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "lodemon-765d5f4748-dps7q" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "instance": "10.142.0.17:9100" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "instance": "10.142.0.61:9100" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 04:47:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468057" 04:47:37 INFO [http_cmd]: http status code OK 04:47:37 DEBUG --- status code --- 04:47:37 DEBUG http status code is 200 (expected 200) 04:47:37 DEBUG --- http response --- 04:47:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:37 DEBUG ----- output ----- 04:47:37 DEBUG { 04:47:37 DEBUG "status": "success", 04:47:37 DEBUG "data": { 04:47:37 DEBUG "resultType": "vector", 04:47:37 DEBUG "result": [ 04:47:37 DEBUG { 04:47:37 DEBUG "metric": { 04:47:37 DEBUG "pod": "ds-idrepo-0" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG [...] 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG 04:47:37 DEBUG "pod": "ds-idrepo-2" 04:47:37 DEBUG }, 04:47:37 DEBUG "value": [ 04:47:37 DEBUG 1684468057, 04:47:37 DEBUG "0" 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG ] 04:47:37 DEBUG } 04:47:37 DEBUG } 127.0.0.1 - - [19/May/2023 04:47:43] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:47:49] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 04:47:51 INFO 04:47:51 INFO [loop_until]: kubectl --namespace=xlou top pods 04:47:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:47:51 INFO [loop_until]: OK (rc = 0) 04:47:51 DEBUG --- stdout --- 04:47:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 5Mi am-7849cf7bdb-776xr 524m 9131Mi am-7849cf7bdb-jcbb8 495m 9111Mi am-7849cf7bdb-zrhp7 503m 9148Mi ds-cts-0 1936m 13773Mi ds-cts-1 1962m 13736Mi ds-cts-2 1950m 13706Mi ds-idrepo-0 655m 13777Mi ds-idrepo-1 13m 13784Mi ds-idrepo-2 14m 13786Mi end-user-ui-787cb4f6b4-56bl4 1m 5Mi idm-78f4b47cb9-bpwjf 7m 3277Mi idm-78f4b47cb9-vh9b7 5m 3260Mi lodemon-765d5f4748-dps7q 21m 70Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 1241m 700Mi 04:47:51 DEBUG --- stderr --- 04:47:51 DEBUG 04:47:56 INFO 04:47:56 INFO [loop_until]: kubectl --namespace=xlou top node 04:47:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:47:56 INFO [loop_until]: OK (rc = 0) 04:47:56 DEBUG --- stdout --- 04:47:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 602m 3% 11323Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 76m 0% 4902Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 633m 3% 11397Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 642m 4% 11758Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 161m 1% 3095Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 103m 0% 1337Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5174Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14770Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 1844m 11% 14562Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 65m 0% 14835Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 625m 3% 14766Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 1968m 12% 14573Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2007m 12% 14637Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 1270m 7% 2770Mi 4% 127.0.0.1 - - [19/May/2023 04:47:56] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 04:47:56 DEBUG --- stderr --- 04:47:56 DEBUG 127.0.0.1 - - [19/May/2023 04:48:03] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:09] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:16] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:17] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:24] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:24] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:31] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:32] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "instance": "10.142.0.17:9100" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "instance": "10.142.0.61:9100" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0.001333333333333068" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "ds-idrepo-0" 04:48:37 DEBUG "pod": "ds-idrepo-0" 04:48:37 DEBUG }, 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "ds-idrepo-2" 04:48:37 DEBUG "pod": "ds-idrepo-2" 04:48:37 DEBUG }, 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684468117" 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "ds-cts-2" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "ds-idrepo-2" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "14252654592" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "instance": "10.142.0.17:9100" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "instance": "10.142.0.61:9100" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "5889.4" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG --- http response --- 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468117" 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:48:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:48:37 DEBUG }, 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:48:37 DEBUG "pod": "idm-78f4b47cb9-vh9b7" 04:48:37 DEBUG }, 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG "100548803.59957552" 04:48:37 DEBUG ] 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "instance": "10.142.0.17:9100" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "instance": "10.142.0.61:9100" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "8948.266666666666" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "ds-cts-2" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "ds-idrepo-2" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "idm-78f4b47cb9-vh9b7" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0.0028803090337369205" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "ds-cts-2" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "ds-idrepo-2" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0.2" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "1.0345598812075958" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "idm-78f4b47cb9-vh9b7" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "idm-78f4b47cb9-vh9b7" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "instance": "10.142.0.17:9100" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "instance": "10.142.0.61:9100" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0.00033333333333350387" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 04:48:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468117" 04:48:37 INFO [http_cmd]: http status code OK 04:48:37 DEBUG --- status code --- 04:48:37 DEBUG http status code is 200 (expected 200) 04:48:37 DEBUG --- http response --- 04:48:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:37 DEBUG ----- output ----- 04:48:37 DEBUG { 04:48:37 DEBUG "status": "success", 04:48:37 DEBUG "data": { 04:48:37 DEBUG "resultType": "vector", 04:48:37 DEBUG "result": [ 04:48:37 DEBUG { 04:48:37 DEBUG "metric": { 04:48:37 DEBUG "pod": "ds-idrepo-0" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG [...] 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG 04:48:37 DEBUG "pod": "ds-idrepo-2" 04:48:37 DEBUG }, 04:48:37 DEBUG "value": [ 04:48:37 DEBUG 1684468117, 04:48:37 DEBUG "0" 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG ] 04:48:37 DEBUG } 04:48:37 DEBUG } 127.0.0.1 - - [19/May/2023 04:48:39] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:39] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:46] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:47] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 04:48:51 INFO 04:48:51 INFO [loop_until]: kubectl --namespace=xlou top pods 04:48:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:48:51 INFO [loop_until]: OK (rc = 0) 04:48:51 DEBUG --- stdout --- 04:48:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 5Mi am-7849cf7bdb-776xr 523m 9131Mi am-7849cf7bdb-jcbb8 486m 9110Mi am-7849cf7bdb-zrhp7 525m 9149Mi ds-cts-0 2322m 13705Mi ds-cts-1 2766m 13752Mi ds-cts-2 2190m 13388Mi ds-idrepo-0 585m 13777Mi ds-idrepo-1 14m 13784Mi ds-idrepo-2 16m 13786Mi end-user-ui-787cb4f6b4-56bl4 1m 5Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 5m 3260Mi lodemon-765d5f4748-dps7q 22m 70Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 707m 874Mi 04:48:51 DEBUG --- stderr --- 04:48:51 DEBUG 127.0.0.1 - - [19/May/2023 04:48:53] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:48:54] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 04:48:56 INFO 04:48:56 INFO [loop_until]: kubectl --namespace=xlou top node 04:48:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:48:56 INFO [loop_until]: OK (rc = 0) 04:48:56 DEBUG --- stdout --- 04:48:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 610m 3% 11323Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4902Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 657m 4% 11397Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 634m 3% 11750Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 157m 0% 3085Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 124m 0% 1339Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5174Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14770Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 2319m 14% 14459Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 71m 0% 14847Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 601m 3% 14763Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 2274m 14% 14253Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2591m 16% 14577Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 1081m 6% 2932Mi 5% 04:48:56 DEBUG --- stderr --- 04:48:56 DEBUG 127.0.0.1 - - [19/May/2023 04:49:01] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:02] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:08] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:09] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:16] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:17] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:23] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:24] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:31] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:32] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:33] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "instance": "10.142.0.17:9100" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "instance": "10.142.0.61:9100" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0.0013999999999981581" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "ds-idrepo-0" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "ds-idrepo-2" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "instance": "10.142.0.17:9100" 04:49:37 DEBUG "pod": "ds-cts-2" 04:49:37 DEBUG }, 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "instance": "10.142.0.61:9100" 04:49:37 DEBUG "pod": "ds-idrepo-2" 04:49:37 DEBUG }, 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "9689.433333333332" 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684468177" 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "ds-idrepo-0" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "ds-idrepo-2" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG } 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG ] 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG } 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG } 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "14201217024" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "129397628.29250535" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG --- http response --- 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468177" 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "ds-cts-2" 04:49:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:49:37 DEBUG }, 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "ds-idrepo-2" 04:49:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:49:37 DEBUG }, 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0" 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "instance": "10.142.0.17:9100" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "instance": "10.142.0.61:9100" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "19548.73333333333" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "1.3672313772737867" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0.2" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "ds-cts-2" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "ds-idrepo-2" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "1.2462090342434085" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0.19172927564291553" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468177" 04:49:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468177" 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "instance": "10.142.0.17:9100" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "instance": "10.142.0.61:9100" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0.0003333333333333854" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 04:49:37 INFO [http_cmd]: http status code OK 04:49:37 DEBUG --- status code --- 04:49:37 DEBUG http status code is 200 (expected 200) 04:49:37 DEBUG --- http response --- 04:49:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:37 DEBUG ----- output ----- 04:49:37 DEBUG { 04:49:37 DEBUG "status": "success", 04:49:37 DEBUG "data": { 04:49:37 DEBUG "resultType": "vector", 04:49:37 DEBUG "result": [ 04:49:37 DEBUG { 04:49:37 DEBUG "metric": { 04:49:37 DEBUG "pod": "ds-idrepo-0" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG [...] 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG 04:49:37 DEBUG "pod": "ds-idrepo-2" 04:49:37 DEBUG }, 04:49:37 DEBUG "value": [ 04:49:37 DEBUG 1684468177, 04:49:37 DEBUG "0" 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG ] 04:49:37 DEBUG } 04:49:37 DEBUG } 127.0.0.1 - - [19/May/2023 04:49:39] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:40] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:41] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:47] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:48] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:49] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 04:49:51 INFO 04:49:51 INFO [loop_until]: kubectl --namespace=xlou top pods 04:49:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:49:51 INFO [loop_until]: OK (rc = 0) 04:49:51 DEBUG --- stdout --- 04:49:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 5Mi am-7849cf7bdb-776xr 515m 9131Mi am-7849cf7bdb-jcbb8 504m 9110Mi am-7849cf7bdb-zrhp7 516m 9148Mi ds-cts-0 2376m 13588Mi ds-cts-1 1844m 13701Mi ds-cts-2 1963m 13520Mi ds-idrepo-0 539m 13777Mi ds-idrepo-1 12m 13784Mi ds-idrepo-2 15m 13786Mi end-user-ui-787cb4f6b4-56bl4 1m 5Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 5m 3260Mi lodemon-765d5f4748-dps7q 24m 70Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 344m 824Mi 04:49:51 DEBUG --- stderr --- 04:49:51 DEBUG 127.0.0.1 - - [19/May/2023 04:49:56] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 04:49:56 INFO 04:49:56 INFO [loop_until]: kubectl --namespace=xlou top node 04:49:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:49:56 INFO [loop_until]: OK (rc = 0) 04:49:56 DEBUG --- stdout --- 04:49:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 604m 3% 11319Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 76m 0% 4903Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 646m 4% 11396Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 644m 4% 11755Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 167m 1% 3101Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 125m 0% 1335Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5174Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 70m 0% 14767Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 2391m 15% 14566Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 68m 0% 14838Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 631m 3% 14764Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 2240m 14% 14353Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2286m 14% 14459Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 726m 4% 3092Mi 5% 04:49:56 DEBUG --- stderr --- 04:49:56 DEBUG 127.0.0.1 - - [19/May/2023 04:49:57] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:49:58] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:04] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:05] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:06] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:12] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:13] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:14] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:21] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:22] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:23] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:29] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:30] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:31] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:32] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468237" 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "instance": "10.142.0.17:9100" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "instance": "10.142.0.61:9100" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0.0007999999999990829" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG { 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG "data": { 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:37 DEBUG "result": [ 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "pod": "ds-idrepo-0" 04:50:37 DEBUG "data": { 04:50:37 DEBUG }, 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "value": [ 04:50:37 DEBUG "result": [ 04:50:37 DEBUG 04:50:37 DEBUG { 04:50:37 DEBUG 04:50:37 DEBUG "metric": { 04:50:37 DEBUG 04:50:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468237" 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 DEBUG [...] 04:50:37 DEBUG }, 04:50:37 DEBUG 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "ds-idrepo-2" 04:50:37 DEBUG 04:50:37 DEBUG }, 04:50:37 DEBUG [...] 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG 04:50:37 DEBUG "0" 04:50:37 DEBUG 04:50:37 DEBUG ] 04:50:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:50:37 DEBUG } 04:50:37 DEBUG }, 04:50:37 DEBUG ] 04:50:37 DEBUG "value": [ 04:50:37 DEBUG } 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG } 04:50:37 DEBUG "0" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "ds-cts-2" 04:50:37 DEBUG "instance": "10.142.0.17:9100" 04:50:37 DEBUG }, 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "ds-idrepo-2" 04:50:37 DEBUG "instance": "10.142.0.61:9100" 04:50:37 DEBUG }, 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG "value": [ 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0" 04:50:37 DEBUG "7228.766666666667" 04:50:37 DEBUG ] 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684468237" 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:50:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:50:37 DEBUG }, 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:50:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:50:37 DEBUG }, 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0" 04:50:37 DEBUG "0" 04:50:37 DEBUG ] 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 DEBUG ----- output ----- 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468237" 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 DEBUG { 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "ds-idrepo-0" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "ds-idrepo-2" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "14236729344" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG { 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG "data": { 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "data": { 04:50:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG }, 04:50:37 DEBUG "result": [ 04:50:37 DEBUG "value": [ 04:50:37 DEBUG { 04:50:37 DEBUG 04:50:37 DEBUG "metric": { 04:50:37 DEBUG 04:50:37 DEBUG "instance": "10.142.0.17:9100" 04:50:37 DEBUG 04:50:37 DEBUG }, 04:50:37 DEBUG [...] 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:50:37 DEBUG [...] 04:50:37 DEBUG }, 04:50:37 DEBUG 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG 04:50:37 DEBUG "0.2" 04:50:37 DEBUG "instance": "10.142.0.61:9100" 04:50:37 DEBUG ] 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG }, 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG } 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG "value": [ 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG ] 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG } 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG "19836.966666666667" 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG } 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:37 DEBUG ] 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG } 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG ] 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG } 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG } 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:50:37 DEBUG "pod": "ds-cts-2" 04:50:37 DEBUG }, 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:50:37 DEBUG "pod": "ds-idrepo-2" 04:50:37 DEBUG }, 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0" 04:50:37 DEBUG "0" 04:50:37 DEBUG ] 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684468237" 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0.705592300450852" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "ds-cts-2" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "ds-idrepo-2" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "admin-ui-8666f85968-92rxx" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "overseer-0-6d59df6649-vlgh6" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0.1915911739611181" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468237" 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "instance": "10.142.0.17:9100" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "instance": "10.142.0.61:9100" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0.0003333333333333558" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 04:50:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468237" 04:50:37 INFO [http_cmd]: http status code OK 04:50:37 DEBUG --- status code --- 04:50:37 DEBUG http status code is 200 (expected 200) 04:50:37 DEBUG --- http response --- 04:50:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:37 DEBUG ----- output ----- 04:50:37 DEBUG { 04:50:37 DEBUG "status": "success", 04:50:37 DEBUG "data": { 04:50:37 DEBUG "resultType": "vector", 04:50:37 DEBUG "result": [ 04:50:37 DEBUG { 04:50:37 DEBUG "metric": { 04:50:37 DEBUG "pod": "ds-idrepo-0" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG [...] 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG 04:50:37 DEBUG "pod": "ds-idrepo-2" 04:50:37 DEBUG }, 04:50:37 DEBUG "value": [ 04:50:37 DEBUG 1684468237, 04:50:37 DEBUG "0" 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG ] 04:50:37 DEBUG } 04:50:37 DEBUG } 127.0.0.1 - - [19/May/2023 04:50:38] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:39] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:40] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:41] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:48] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:48] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:49] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:50] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 04:50:51 INFO 04:50:51 INFO [loop_until]: kubectl --namespace=xlou top pods 04:50:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:50:51 INFO [loop_until]: OK (rc = 0) 04:50:51 DEBUG --- stdout --- 04:50:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 5Mi am-7849cf7bdb-776xr 515m 9132Mi am-7849cf7bdb-jcbb8 503m 9110Mi am-7849cf7bdb-zrhp7 530m 9149Mi ds-cts-0 2248m 13605Mi ds-cts-1 2158m 13536Mi ds-cts-2 2043m 13542Mi ds-idrepo-0 571m 13777Mi ds-idrepo-1 12m 13784Mi ds-idrepo-2 16m 13786Mi end-user-ui-787cb4f6b4-56bl4 1m 5Mi idm-78f4b47cb9-bpwjf 7m 3278Mi idm-78f4b47cb9-vh9b7 5m 3266Mi lodemon-765d5f4748-dps7q 39m 70Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 331m 949Mi 04:50:51 DEBUG --- stderr --- 04:50:51 DEBUG 04:50:56 INFO 04:50:56 INFO [loop_until]: kubectl --namespace=xlou top node 04:50:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:50:56 INFO [loop_until]: OK (rc = 0) 04:50:56 DEBUG --- stdout --- 04:50:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 610m 3% 11323Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 4907Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 643m 4% 11399Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 639m 4% 11754Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 163m 1% 3103Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 137m 0% 1337Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 93m 0% 5174Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14769Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 2104m 13% 14386Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 67m 0% 14838Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 443m 2% 14763Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 2147m 13% 14392Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2249m 14% 14480Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 736m 4% 3305Mi 5% 04:50:56 DEBUG --- stderr --- 04:50:56 DEBUG 127.0.0.1 - - [19/May/2023 04:50:57] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:58] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:58] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:50:59] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:06] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:07] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:08] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:09] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:15] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:16] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:17] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:18] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:24] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:25] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:26] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:27] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:34] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:34] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:35] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:36] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:37] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "instance": "10.142.0.17:9100" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "instance": "10.142.0.61:9100" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0.0006333333333335152" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468297" 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG { 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "data": { 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG "result": [ 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG { 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:51:37 DEBUG "pod": "ds-idrepo-0" 04:51:37 DEBUG "instance": "10.142.0.17:9100" 04:51:37 DEBUG }, 04:51:37 DEBUG }, 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG "value": [ 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG [...] 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:51:37 DEBUG "pod": "ds-idrepo-2" 04:51:37 DEBUG "instance": "10.142.0.61:9100" 04:51:37 DEBUG }, 04:51:37 DEBUG }, 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG "value": [ 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0" 04:51:37 DEBUG "0" 04:51:37 DEBUG "6882.099999999999" 04:51:37 DEBUG ] 04:51:37 DEBUG ] 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG ] 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468297" 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG { 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG "data": { 04:51:37 DEBUG { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG "data": { 04:51:37 DEBUG { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "result": [ 04:51:37 DEBUG "pod": "ds-cts-2" 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 DEBUG { 04:51:37 DEBUG }, 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "value": [ 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG "instance": "10.142.0.17:9100" 04:51:37 DEBUG 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG }, 04:51:37 DEBUG 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684468297" 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG { 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG 04:51:37 DEBUG "pod": "ds-idrepo-2" 04:51:37 DEBUG "data": { 04:51:37 DEBUG 04:51:37 DEBUG }, 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG 04:51:37 DEBUG "value": [ 04:51:37 DEBUG "result": [ 04:51:37 DEBUG "instance": "10.142.0.61:9100" 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG { 04:51:37 DEBUG }, 04:51:37 DEBUG "0" 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "value": [ 04:51:37 DEBUG ] 04:51:37 DEBUG "pod": "ds-cts-1" 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG } 04:51:37 DEBUG }, 04:51:37 DEBUG "19235" 04:51:37 DEBUG ] 04:51:37 DEBUG "value": [ 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG 04:51:37 DEBUG ] 04:51:37 DEBUG 04:51:37 DEBUG } 04:51:37 DEBUG [...] 04:51:37 DEBUG } 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG }, 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG "value": [ 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG "3467.3048887504215" 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:51:37 DEBUG ] 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG } 04:51:37 DEBUG { 04:51:37 DEBUG ] 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG } 04:51:37 DEBUG "data": { 04:51:37 DEBUG } 04:51:37 DEBUG "resultType": "vector", 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "ds-cts-1" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0.35567336505194486" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "ds-idrepo-0" 04:51:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:51:37 DEBUG }, 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "ds-idrepo-2" 04:51:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:51:37 DEBUG }, 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0" 04:51:37 DEBUG "14339031040" 04:51:37 DEBUG ] 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:51:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:51:37 DEBUG }, 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:51:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG }, 04:51:37 DEBUG }, 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:51:37 DEBUG "value": [ 04:51:37 DEBUG "value": [ 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG { 04:51:37 DEBUG "0.2" 04:51:37 DEBUG "0" 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG ] 04:51:37 DEBUG ] 04:51:37 DEBUG "data": { 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG ] 04:51:37 DEBUG ] 04:51:37 DEBUG "result": [ 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG { 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "ds-cts-2" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "ds-idrepo-2" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG "0" 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG ] 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG } 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG ] 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:37 DEBUG } 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG } 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0.7656987316585988" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "ds-cts-1" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "ds-cts-2" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "ds-idrepo-2" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "ds-cts-1" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "instance": "10.142.0.17:9100" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "instance": "10.142.0.61:9100" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0.0003333333333330302" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 04:51:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468297" 04:51:37 INFO [http_cmd]: http status code OK 04:51:37 DEBUG --- status code --- 04:51:37 DEBUG http status code is 200 (expected 200) 04:51:37 DEBUG --- http response --- 04:51:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:37 DEBUG ----- output ----- 04:51:37 DEBUG { 04:51:37 DEBUG "status": "success", 04:51:37 DEBUG "data": { 04:51:37 DEBUG "resultType": "vector", 04:51:37 DEBUG "result": [ 04:51:37 DEBUG { 04:51:37 DEBUG "metric": { 04:51:37 DEBUG "pod": "ds-idrepo-0" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG [...] 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG 04:51:37 DEBUG "pod": "ds-idrepo-2" 04:51:37 DEBUG }, 04:51:37 DEBUG "value": [ 04:51:37 DEBUG 1684468297, 04:51:37 DEBUG "0" 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG ] 04:51:37 DEBUG } 04:51:37 DEBUG } 127.0.0.1 - - [19/May/2023 04:51:44] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:45] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:45] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:46] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:47] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 04:51:51 INFO 04:51:51 INFO [loop_until]: kubectl --namespace=xlou top pods 04:51:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:51:51 INFO [loop_until]: OK (rc = 0) 04:51:51 DEBUG --- stdout --- 04:51:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 5Mi am-7849cf7bdb-776xr 527m 9137Mi am-7849cf7bdb-jcbb8 475m 9110Mi am-7849cf7bdb-zrhp7 516m 9148Mi ds-cts-0 2253m 13433Mi ds-cts-1 2451m 13624Mi ds-cts-2 1822m 13636Mi ds-idrepo-0 404m 13777Mi ds-idrepo-1 13m 13785Mi ds-idrepo-2 14m 13786Mi end-user-ui-787cb4f6b4-56bl4 1m 5Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 5m 3266Mi lodemon-765d5f4748-dps7q 44m 70Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 695m 1523Mi 04:51:51 DEBUG --- stderr --- 04:51:51 DEBUG 127.0.0.1 - - [19/May/2023 04:51:54] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:55] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:55] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:51:56] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 04:51:56 INFO 04:51:56 INFO [loop_until]: kubectl --namespace=xlou top node 04:51:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:51:56 INFO [loop_until]: OK (rc = 0) 04:51:56 DEBUG --- stdout --- 04:51:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 598m 3% 11319Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 77m 0% 4908Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 660m 4% 11407Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 628m 3% 11756Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 156m 0% 3111Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 142m 0% 1332Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5171Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14766Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 2362m 14% 14486Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 66m 0% 14838Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 448m 2% 14764Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 1799m 11% 14490Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2087m 13% 14314Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 698m 4% 3516Mi 5% 04:51:56 DEBUG --- stderr --- 04:51:56 DEBUG 127.0.0.1 - - [19/May/2023 04:51:57] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:04] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:05] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:05] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:06] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:07] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:14] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:15] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:16] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:16] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:17] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:24] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:25] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:26] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:27] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:27] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:34] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:35] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:36] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:36] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG { 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "data": { 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "result": [ 04:52:37 DEBUG { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "instance": "10.142.0.17:9100" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG [...] 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "instance": "10.142.0.61:9100" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG "0.001166666666666553" 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG { 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "data": { 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "result": [ 04:52:37 DEBUG { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "instance": "10.142.0.17:9100" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG [...] 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "instance": "10.142.0.61:9100" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG "7509.6" 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG { 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "data": { 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "result": [ 04:52:37 DEBUG { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "pod": "ds-idrepo-0" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG [...] 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "pod": "ds-idrepo-2" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG "0" 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG { 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "data": { 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "result": [ 04:52:37 DEBUG { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG [...] 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG "0" 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 127.0.0.1 - - [19/May/2023 04:52:37] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684468357" 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- status code --- 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG "data": { 04:52:37 DEBUG "data": { 04:52:37 DEBUG "data": { 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:52:37 DEBUG "data": { 04:52:37 DEBUG "data": { 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:52:37 DEBUG "data": { 04:52:37 DEBUG "data": { 04:52:37 DEBUG { 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "data": { 04:52:37 DEBUG "data": { 04:52:37 DEBUG "data": { 04:52:37 DEBUG "data": { 04:52:37 DEBUG "result": [ 04:52:37 DEBUG "result": [ 04:52:37 DEBUG "result": [ 04:52:37 DEBUG { 04:52:37 DEBUG "result": [ 04:52:37 DEBUG "result": [ 04:52:37 DEBUG { 04:52:37 DEBUG "result": [ 04:52:37 DEBUG "result": [ 04:52:37 DEBUG "data": { 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "result": [ 04:52:37 DEBUG "result": [ 04:52:37 DEBUG "result": [ 04:52:37 DEBUG "result": [ 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "data": { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "data": { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "result": [ 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG { 04:52:37 DEBUG "instance": "10.142.0.17:9100" 04:52:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:52:37 DEBUG "pod": "ds-cts-2" 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:52:37 DEBUG "pod": "ds-cts-1" 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "pod": "ds-cts-1" 04:52:37 DEBUG "pod": "ds-idrepo-0" 04:52:37 DEBUG { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG "result": [ 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG "result": [ 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "pod": "ds-cts-2" 04:52:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:52:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:52:37 DEBUG "pod": "ds-cts-2" 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG { 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG { 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "metric": { 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "metric": { 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "pod": "ds-cts-1" 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG }, 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG }, 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG [...] 04:52:37 DEBUG [...] 04:52:37 DEBUG [...] 04:52:37 DEBUG "value": [ 04:52:37 DEBUG [...] 04:52:37 DEBUG [...] 04:52:37 DEBUG "value": [ 04:52:37 DEBUG [...] 04:52:37 DEBUG [...] 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG [...] 04:52:37 DEBUG [...] 04:52:37 DEBUG [...] 04:52:37 DEBUG [...] 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG [...] 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "instance": "10.142.0.61:9100" 04:52:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:52:37 DEBUG "pod": "ds-idrepo-2" 04:52:37 DEBUG [...] 04:52:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:52:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:52:37 DEBUG [...] 04:52:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:52:37 DEBUG "pod": "ds-idrepo-2" 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG 04:52:37 DEBUG "pod": "ds-idrepo-2" 04:52:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:52:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:52:37 DEBUG "pod": "ds-idrepo-2" 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG }, 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "value": [ 04:52:37 DEBUG "20500.033333333333" 04:52:37 DEBUG "0.2" 04:52:37 DEBUG "0" 04:52:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:52:37 DEBUG "14244106240" 04:52:37 DEBUG "0" 04:52:37 DEBUG "pod": "login-ui-57fddf97c8-hrrhn" 04:52:37 DEBUG "0" 04:52:37 DEBUG "0" 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG }, 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG }, 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG "0" 04:52:37 DEBUG "0" 04:52:37 DEBUG "0" 04:52:37 DEBUG "0" 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG "value": [ 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG "value": [ 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG "0.7158979697354972" 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG "0" 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG "0" 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468357" 04:52:37 INFO [http_cmd]: http status code OK 04:52:37 DEBUG --- status code --- 04:52:37 DEBUG http status code is 200 (expected 200) 04:52:37 DEBUG --- http response --- 04:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:52:37 DEBUG ----- output ----- 04:52:37 DEBUG { 04:52:37 DEBUG "status": "success", 04:52:37 DEBUG "data": { 04:52:37 DEBUG "resultType": "vector", 04:52:37 DEBUG "result": [ 04:52:37 DEBUG { 04:52:37 DEBUG "metric": { 04:52:37 DEBUG "pod": "ds-cts-1" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG [...] 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG 04:52:37 DEBUG "pod": "login-ui-57fddf97c8-hrrhn" 04:52:37 DEBUG }, 04:52:37 DEBUG "value": [ 04:52:37 DEBUG 1684468357, 04:52:37 DEBUG "0" 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG ] 04:52:37 DEBUG } 04:52:37 DEBUG } 04:52:38 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468358" 04:52:38 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468358" 04:52:38 INFO [http_cmd]: http status code OK 04:52:38 DEBUG --- status code --- 04:52:38 DEBUG http status code is 200 (expected 200) 04:52:38 DEBUG --- http response --- 04:52:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:52:38 DEBUG ----- output ----- 04:52:38 DEBUG { 04:52:38 DEBUG "status": "success", 04:52:38 DEBUG "data": { 04:52:38 DEBUG "resultType": "vector", 04:52:38 DEBUG "result": [ 04:52:38 DEBUG { 04:52:38 DEBUG "metric": { 04:52:38 DEBUG "pod": "ds-idrepo-0" 04:52:38 DEBUG }, 04:52:38 DEBUG "value": [ 04:52:38 DEBUG 04:52:38 DEBUG 04:52:38 DEBUG 04:52:38 DEBUG [...] 04:52:38 DEBUG 04:52:38 DEBUG 04:52:38 DEBUG 04:52:38 DEBUG "pod": "ds-idrepo-2" 04:52:38 DEBUG }, 04:52:38 DEBUG "value": [ 04:52:38 DEBUG 1684468358, 04:52:38 DEBUG "0" 04:52:38 DEBUG ] 04:52:38 DEBUG } 04:52:38 DEBUG ] 04:52:38 DEBUG } 04:52:38 DEBUG } 04:52:38 INFO [http_cmd]: http status code OK 04:52:38 DEBUG --- status code --- 04:52:38 DEBUG http status code is 200 (expected 200) 04:52:38 DEBUG --- http response --- 04:52:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:52:38 DEBUG ----- output ----- 04:52:38 DEBUG { 04:52:38 DEBUG "status": "success", 04:52:38 DEBUG "data": { 04:52:38 DEBUG "resultType": "vector", 04:52:38 DEBUG "result": [ 04:52:38 DEBUG { 04:52:38 DEBUG "metric": { 04:52:38 DEBUG "instance": "10.142.0.17:9100" 04:52:38 DEBUG }, 04:52:38 DEBUG "value": [ 04:52:38 DEBUG 04:52:38 DEBUG 04:52:38 DEBUG 04:52:38 DEBUG [...] 04:52:38 DEBUG 04:52:38 DEBUG 04:52:38 DEBUG 04:52:38 DEBUG "instance": "10.142.0.61:9100" 04:52:38 DEBUG }, 04:52:38 DEBUG "value": [ 04:52:38 DEBUG 1684468358, 04:52:38 DEBUG "0" 04:52:38 DEBUG ] 04:52:38 DEBUG } 04:52:38 DEBUG ] 04:52:38 DEBUG } 04:52:38 DEBUG } 127.0.0.1 - - [19/May/2023 04:52:38] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:45] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:46] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:47] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:47] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:48] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:49] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 04:52:52 INFO 04:52:52 INFO [loop_until]: kubectl --namespace=xlou top pods 04:52:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:52:52 INFO [loop_until]: OK (rc = 0) 04:52:52 DEBUG --- stdout --- 04:52:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 5Mi am-7849cf7bdb-776xr 524m 9136Mi am-7849cf7bdb-jcbb8 492m 9110Mi am-7849cf7bdb-zrhp7 502m 9148Mi ds-cts-0 1709m 13568Mi ds-cts-1 2264m 13632Mi ds-cts-2 2186m 13529Mi ds-idrepo-0 384m 13777Mi ds-idrepo-1 13m 13785Mi ds-idrepo-2 15m 13786Mi end-user-ui-787cb4f6b4-56bl4 1m 5Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 5m 3266Mi lodemon-765d5f4748-dps7q 51m 70Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 762m 1784Mi 04:52:52 DEBUG --- stderr --- 04:52:52 DEBUG 127.0.0.1 - - [19/May/2023 04:52:56] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 04:52:56 INFO 04:52:56 INFO [loop_until]: kubectl --namespace=xlou top node 04:52:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:52:56 INFO [loop_until]: OK (rc = 0) 04:52:56 DEBUG --- stdout --- 04:52:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 595m 3% 11323Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 73m 0% 4908Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 654m 4% 11402Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 619m 3% 11757Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 156m 0% 3103Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 156m 0% 1336Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5170Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 67m 0% 14770Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 1970m 12% 14478Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 65m 0% 14839Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 446m 2% 14763Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 2255m 14% 14399Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 1857m 11% 14433Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 704m 4% 3777Mi 6% 04:52:56 DEBUG --- stderr --- 04:52:56 DEBUG 127.0.0.1 - - [19/May/2023 04:52:57] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:58] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:59] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:52:59] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:00] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:07] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:08] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:09] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:09] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:10] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:11] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:18] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:19] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:20] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:20] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:21] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:22] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:29] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:30] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:31] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:31] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:32] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:33] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG { 04:53:37 DEBUG { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "data": { 04:53:37 DEBUG "data": { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "result": [ 04:53:37 DEBUG "result": [ 04:53:37 DEBUG { 04:53:37 DEBUG { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "instance": "10.142.0.17:9100" 04:53:37 DEBUG "instance": "10.142.0.17:9100" 04:53:37 DEBUG }, 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG 04:53:37 DEBUG "instance": "10.142.0.61:9100" 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:53:37 DEBUG "instance": "10.142.0.61:9100" 04:53:37 DEBUG }, 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG { 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "0.0012000000000000456" 04:53:37 DEBUG "data": { 04:53:37 DEBUG "7315.166666666666" 04:53:37 DEBUG ] 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG "result": [ 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG { 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG "metric": { 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG "pod": "ds-idrepo-0" 04:53:37 DEBUG } 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "ds-idrepo-2" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "0" 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "data": { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "result": [ 04:53:37 DEBUG { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "0" 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "data": { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "result": [ 04:53:37 DEBUG { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "0" 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- status code --- 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "data": { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "result": [ 04:53:37 DEBUG { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "pod": "ds-cts-1" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "296500213.7669915" 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "data": { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "result": [ 04:53:37 DEBUG { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "pod": "ds-cts-1" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "0" 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "data": { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "result": [ 04:53:37 DEBUG { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "pod": "ds-cts-2" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "ds-idrepo-2" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "0" 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684468417" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- status code --- 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- http response --- 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG { 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG ----- output ----- 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG { 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "data": { 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG "data": { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG "data": { 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "result": [ 04:53:37 DEBUG { 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:53:37 DEBUG { 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:53:37 DEBUG { 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:53:37 DEBUG "result": [ 04:53:37 DEBUG { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG "result": [ 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "data": { 04:53:37 DEBUG { 04:53:37 DEBUG { 04:53:37 DEBUG { 04:53:37 DEBUG "data": { 04:53:37 DEBUG { 04:53:37 DEBUG "data": { 04:53:37 DEBUG { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "pod": "ds-idrepo-0" 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "instance": "10.142.0.17:9100" 04:53:37 DEBUG }, 04:53:37 DEBUG "result": [ 04:53:37 DEBUG "data": { 04:53:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:53:37 DEBUG "data": { 04:53:37 DEBUG "result": [ 04:53:37 DEBUG "data": { 04:53:37 DEBUG "result": [ 04:53:37 DEBUG "data": { 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG }, 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "result": [ 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "result": [ 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "result": [ 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "result": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "ds-cts-2" 04:53:37 DEBUG { 04:53:37 DEBUG 04:53:37 DEBUG { 04:53:37 DEBUG "pod": "ds-cts-1" 04:53:37 DEBUG { 04:53:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:53:37 DEBUG { 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG }, 04:53:37 DEBUG "metric": { 04:53:37 DEBUG 04:53:37 DEBUG "metric": { 04:53:37 DEBUG }, 04:53:37 DEBUG "metric": { 04:53:37 DEBUG }, 04:53:37 DEBUG "metric": { 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "pod": "ds-cts-2" 04:53:37 DEBUG 04:53:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG }, 04:53:37 DEBUG [...] 04:53:37 DEBUG }, 04:53:37 DEBUG 04:53:37 DEBUG }, 04:53:37 DEBUG 04:53:37 DEBUG }, 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "ds-idrepo-2" 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG "instance": "10.142.0.61:9100" 04:53:37 DEBUG }, 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG }, 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "0" 04:53:37 DEBUG "pod": "ds-idrepo-2" 04:53:37 DEBUG 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG 04:53:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:53:37 DEBUG 04:53:37 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 04:53:37 DEBUG 04:53:37 DEBUG "20204.066666666666" 04:53:37 DEBUG ] 04:53:37 DEBUG }, 04:53:37 DEBUG 04:53:37 DEBUG "0.2" 04:53:37 DEBUG 04:53:37 DEBUG }, 04:53:37 DEBUG 04:53:37 DEBUG }, 04:53:37 DEBUG 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "pod": "ds-idrepo-2" 04:53:37 DEBUG ] 04:53:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "pod": "am-7849cf7bdb-776xr" 04:53:37 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468417" 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG }, 04:53:37 DEBUG } 04:53:37 DEBUG }, 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG }, 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG }, 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG "0" 04:53:37 DEBUG "value": [ 04:53:37 DEBUG ] 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "0.49451844621694685" 04:53:37 DEBUG "value": [ 04:53:37 DEBUG "14303281152" 04:53:37 DEBUG "value": [ 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG } 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG ] 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG ] 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG "0" 04:53:37 DEBUG } 04:53:37 DEBUG "0" 04:53:37 DEBUG } 04:53:37 DEBUG "0" 04:53:37 DEBUG } 04:53:37 DEBUG "1.0969962457462241" 04:53:37 DEBUG ] 04:53:37 DEBUG ] 04:53:37 DEBUG ] 04:53:37 DEBUG ] 04:53:37 DEBUG ] 04:53:37 DEBUG ] 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:37 INFO [http_cmd]: http status code OK 04:53:37 DEBUG --- status code --- 04:53:37 DEBUG http status code is 200 (expected 200) 04:53:37 DEBUG --- http response --- 04:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:53:37 DEBUG ----- output ----- 04:53:37 DEBUG { 04:53:37 DEBUG "status": "success", 04:53:37 DEBUG "data": { 04:53:37 DEBUG "resultType": "vector", 04:53:37 DEBUG "result": [ 04:53:37 DEBUG { 04:53:37 DEBUG "metric": { 04:53:37 DEBUG "pod": "ds-cts-1" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG [...] 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG 04:53:37 DEBUG "pod": "am-7849cf7bdb-jcbb8" 04:53:37 DEBUG }, 04:53:37 DEBUG "value": [ 04:53:37 DEBUG 1684468417, 04:53:37 DEBUG "0" 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG ] 04:53:37 DEBUG } 04:53:37 DEBUG } 04:53:38 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684468418" 04:53:38 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684468418" 04:53:38 INFO [http_cmd]: http status code OK 04:53:38 DEBUG --- status code --- 04:53:38 DEBUG http status code is 200 (expected 200) 04:53:38 DEBUG --- http response --- 04:53:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:53:38 DEBUG ----- output ----- 04:53:38 DEBUG { 04:53:38 DEBUG "status": "success", 04:53:38 DEBUG "data": { 04:53:38 DEBUG "resultType": "vector", 04:53:38 DEBUG "result": [ 04:53:38 DEBUG { 04:53:38 DEBUG "metric": { 04:53:38 DEBUG "pod": "ds-idrepo-0" 04:53:38 DEBUG }, 04:53:38 DEBUG "value": [ 04:53:38 DEBUG 04:53:38 DEBUG 04:53:38 DEBUG 04:53:38 DEBUG [...] 04:53:38 DEBUG 04:53:38 DEBUG 04:53:38 DEBUG 04:53:38 DEBUG "pod": "ds-idrepo-2" 04:53:38 DEBUG }, 04:53:38 DEBUG "value": [ 04:53:38 DEBUG 1684468418, 04:53:38 DEBUG "0" 04:53:38 DEBUG ] 04:53:38 DEBUG } 04:53:38 DEBUG ] 04:53:38 DEBUG } 04:53:38 DEBUG } 04:53:38 INFO [http_cmd]: http status code OK 04:53:38 DEBUG --- status code --- 04:53:38 DEBUG http status code is 200 (expected 200) 04:53:38 DEBUG --- http response --- 04:53:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:53:38 DEBUG ----- output ----- 04:53:38 DEBUG { 04:53:38 DEBUG "status": "success", 04:53:38 DEBUG "data": { 04:53:38 DEBUG "resultType": "vector", 04:53:38 DEBUG "result": [ 04:53:38 DEBUG { 04:53:38 DEBUG "metric": { 04:53:38 DEBUG "instance": "10.142.0.17:9100" 04:53:38 DEBUG }, 04:53:38 DEBUG "value": [ 04:53:38 DEBUG 04:53:38 DEBUG 04:53:38 DEBUG 04:53:38 DEBUG [...] 04:53:38 DEBUG 04:53:38 DEBUG 04:53:38 DEBUG 04:53:38 DEBUG "instance": "10.142.0.61:9100" 04:53:38 DEBUG }, 04:53:38 DEBUG "value": [ 04:53:38 DEBUG 1684468418, 04:53:38 DEBUG "0.0003333333333333262" 04:53:38 DEBUG ] 04:53:38 DEBUG } 04:53:38 DEBUG ] 04:53:38 DEBUG } 04:53:38 DEBUG } 127.0.0.1 - - [19/May/2023 04:53:40] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:41] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:41] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:42] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:43] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:44] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:45] "GET /monitoring/average?start_time=23-05-19_03:12:40&stop_time=23-05-19_03:29:02 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:51] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 04:53:52 INFO 04:53:52 INFO [loop_until]: kubectl --namespace=xlou top pods 04:53:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:53:52 INFO [loop_until]: OK (rc = 0) 04:53:52 DEBUG --- stdout --- 04:53:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-92rxx 1m 5Mi am-7849cf7bdb-776xr 527m 9136Mi am-7849cf7bdb-jcbb8 493m 9110Mi am-7849cf7bdb-zrhp7 520m 9148Mi ds-cts-0 1981m 13578Mi ds-cts-1 2011m 13562Mi ds-cts-2 1822m 13583Mi ds-idrepo-0 559m 13777Mi ds-idrepo-1 162m 13784Mi ds-idrepo-2 15m 13787Mi end-user-ui-787cb4f6b4-56bl4 1m 5Mi idm-78f4b47cb9-bpwjf 6m 3278Mi idm-78f4b47cb9-vh9b7 5m 3260Mi lodemon-765d5f4748-dps7q 58m 70Mi login-ui-57fddf97c8-hrrhn 1m 3Mi overseer-0-6d59df6649-vlgh6 655m 1995Mi 04:53:52 DEBUG --- stderr --- 04:53:52 DEBUG 127.0.0.1 - - [19/May/2023 04:53:52] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:53] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:54] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:55] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:53:56] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 04:53:57 INFO 127.0.0.1 - - [19/May/2023 04:53:57] "GET /monitoring/average?start_time=23-05-19_03:12:40&stop_time=23-05-19_03:29:02 HTTP/1.1" 200 - 04:53:57 INFO [loop_until]: kubectl --namespace=xlou top node 04:53:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:53:57 INFO [loop_until]: OK (rc = 0) 04:53:57 DEBUG --- stdout --- 04:53:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 591m 3% 11319Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 75m 0% 4909Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 648m 4% 11400Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 602m 3% 11756Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 163m 1% 3112Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 165m 1% 1335Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5166Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 70m 0% 14768Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 1975m 12% 14418Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 241m 1% 14841Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 628m 3% 14766Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 1922m 12% 14453Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 1836m 11% 14448Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 711m 4% 4052Mi 6% 04:53:57 DEBUG --- stderr --- 04:53:57 DEBUG 127.0.0.1 - - [19/May/2023 04:54:03] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:04] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:05] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:06] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:07] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:07] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:08] "GET /monitoring/average?start_time=23-05-19_03:12:40&stop_time=23-05-19_03:29:02 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:15] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:16] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:17] "GET /monitoring/average?start_time=23-05-19_02:07:20&stop_time=23-05-19_02:23:37 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:18] "GET /monitoring/average?start_time=23-05-19_02:23:37&stop_time=23-05-19_02:39:54 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:19] "GET /monitoring/average?start_time=23-05-19_02:39:54&stop_time=23-05-19_02:56:17 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:19] "GET /monitoring/average?start_time=23-05-19_02:56:17&stop_time=23-05-19_03:12:40 HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:20] "GET /monitoring/average?start_time=23-05-19_03:12:40&stop_time=23-05-19_03:29:02 HTTP/1.1" 200 - 04:54:26 INFO Finished: True 04:54:26 INFO Waiting for threads to register finish flag 04:54:57 INFO Done. Have a nice day! :) 127.0.0.1 - - [19/May/2023 04:54:57] "GET /monitoring/stop HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:54:57] "GET /monitoring/average?start_time=23-05-19_01:34:22&stop_time=23-05-19_01:50:51 HTTP/1.1" 200 - 04:55:04 INFO File /tmp/lodemon_data-23-05-18_23:31:41/am_cts_reaper_cache_size.json does not exist. Skipping... 04:55:04 INFO File /tmp/lodemon_data-23-05-18_23:31:41/am_cts_reaper_search_seconds_total.json does not exist. Skipping... 04:55:04 INFO File /tmp/lodemon_data-23-05-18_23:31:41/ds_replication_replica_replayed_updates_conflicts_resolved.json does not exist. Skipping... 04:55:04 INFO File /tmp/lodemon_data-23-05-18_23:31:41/node_disk_read_bytes_total.json does not exist. Skipping... 04:55:04 INFO File /tmp/lodemon_data-23-05-18_23:31:41/node_disk_written_bytes_total.json does not exist. Skipping... 04:55:04 INFO File /tmp/lodemon_data-23-05-18_23:31:41/ds_backend_entry_count.json does not exist. Skipping... 127.0.0.1 - - [19/May/2023 04:55:12] "GET /monitoring/process HTTP/1.1" 200 - 127.0.0.1 - - [19/May/2023 04:55:12] "GET /monitoring/average?start_time=23-05-19_01:50:51&stop_time=23-05-19_02:07:20 HTTP/1.1" 200 -