==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-58bb5f9bf4-7mmfr Namespace: xlou Priority: 0 Node: gke-xlou-cdm-default-pool-cab466d1-h89r/10.142.0.26 Start Time: Fri, 19 May 2023 18:49:35 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=58bb5f9bf4 skaffold.dev/run-id=744b72d8-170c-431d-9195-2319d3359d39 Annotations: Status: Running IP: 10.106.40.208 IPs: IP: 10.106.40.208 Controlled By: ReplicaSet/lodemon-58bb5f9bf4 Containers: lodemon: Container ID: containerd://48e517aa738e773f617373a206b3e779d89ffa472fe3b9436210e17a60b78b20 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:21aa7819571d93a7f3c2e1355b96b08ae4717ae2 Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:b4cd4ad79a4fe740bf283ecfb348b507abe0e3672cdd540fd5aedaa47acc0a7b Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Fri, 19 May 2023 18:49:36 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jjb2d (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-jjb2d: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 08:13:24 INFO 08:13:24 INFO [loop_until]: kubectl --namespace=xlou top pods 08:13:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:13:24 INFO [loop_until]: OK (rc = 0) 08:13:24 DEBUG --- stdout --- 08:13:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5650m 9428Mi am-7849cf7bdb-lxfkl 3985m 9366Mi am-7849cf7bdb-td7js 4457m 9425Mi ds-cts-0 6911m 13884Mi ds-cts-1 7955m 13869Mi ds-cts-2 6815m 13877Mi ds-idrepo-0 2119m 13736Mi ds-idrepo-1 12m 13622Mi ds-idrepo-2 15m 13588Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 8m 3395Mi idm-78f4b47cb9-fmzjg 6m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1190m 834Mi 08:13:24 DEBUG --- stderr --- 08:13:24 DEBUG 08:13:36 INFO 08:13:36 INFO [loop_until]: kubectl --namespace=xlou top node 08:13:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:13:36 INFO [loop_until]: OK (rc = 0) 08:13:36 DEBUG --- stdout --- 08:13:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4377m 27% 11579Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 69m 0% 4951Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6635m 41% 11693Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7435m 46% 12028Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 152m 0% 3116Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 89m 0% 1355Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5311Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 61m 0% 14699Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 7408m 46% 15198Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2143m 13% 14834Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 64m 0% 14632Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 6426m 40% 15174Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 7000m 44% 15201Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1241m 7% 2714Mi 4% 08:13:36 DEBUG --- stderr --- 08:13:36 DEBUG 08:13:48 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566828" 08:13:48 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566828" 08:13:48 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566828" 08:13:48 INFO [http_cmd]: http status code OK 08:13:48 DEBUG --- status code --- 08:13:48 DEBUG http status code is 200 (expected 200) 08:13:48 DEBUG --- http response --- 08:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:13:48 DEBUG ----- output ----- 08:13:48 DEBUG { 08:13:48 DEBUG "status": "success", 08:13:48 DEBUG "data": { 08:13:48 DEBUG "resultType": "vector", 08:13:48 DEBUG "result": [ 08:13:48 DEBUG { 08:13:48 DEBUG "metric": { 08:13:48 DEBUG "pod": "ds-cts-2" 08:13:48 DEBUG }, 08:13:48 DEBUG "value": [ 08:13:48 DEBUG 08:13:48 DEBUG 08:13:48 DEBUG 08:13:48 DEBUG [...] 08:13:48 DEBUG 08:13:48 DEBUG 08:13:48 DEBUG 08:13:48 DEBUG "pod": "ds-idrepo-1" 08:13:48 DEBUG }, 08:13:48 DEBUG "value": [ 08:13:48 DEBUG 1684566828, 08:13:48 DEBUG "0" 08:13:48 DEBUG ] 08:13:48 DEBUG } 08:13:48 DEBUG ] 08:13:48 DEBUG } 08:13:48 DEBUG } 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "ds-cts-2" 08:13:49 DEBUG "pod": "ds-idrepo-2" 08:13:49 DEBUG }, 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "ds-idrepo-1" 08:13:49 DEBUG "pod": "ds-idrepo-1" 08:13:49 DEBUG }, 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566828, 08:13:49 DEBUG 1684566828, 08:13:49 DEBUG "0" 08:13:49 DEBUG "0" 08:13:49 DEBUG ] 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "688.6333333333333" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566829" 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "instance": "10.142.0.17:9100" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "instance": "10.142.0.61:9100" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "6976769.266666667" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "ds-idrepo-1" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "0" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "689.1666666666666" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "ds-idrepo-2" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "ds-idrepo-1" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "0" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "229.66666666666666" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "ds-cts-2" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "ds-idrepo-1" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "0" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566829" 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566829" 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "instance": "10.142.0.17:9100" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "instance": "10.142.0.61:9100" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "0.001266666666666083" 08:13:49 DEBUG ] 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG } 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG ] 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG } 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG } 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "instance": "10.142.0.17:9100" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "instance": "10.142.0.61:9100" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "0.0006666666666667708" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "ds-idrepo-2" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "ds-idrepo-1" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "0" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "0" 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "229.46666666666667" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684566829" 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684566829" 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566829" 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG "data": { 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG { 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG "result": [ 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:13:49 DEBUG { 08:13:49 DEBUG "data": { 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG { 08:13:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG }, 08:13:49 DEBUG { 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG "data": { 08:13:49 DEBUG "value": [ 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "data": { 08:13:49 DEBUG "resultType": "vector", 08:13:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566829" 08:13:49 DEBUG 08:13:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:13:49 DEBUG "resultType": "vector", 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG "result": [ 08:13:49 DEBUG 08:13:49 DEBUG }, 08:13:49 DEBUG "result": [ 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG { 08:13:49 DEBUG 08:13:49 DEBUG "value": [ 08:13:49 DEBUG { 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG "metric": { 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG "metric": { 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "instance": "10.142.0.17:9100" 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:13:49 DEBUG }, 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG }, 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG "value": [ 08:13:49 DEBUG { 08:13:49 DEBUG 08:13:49 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG 08:13:49 DEBUG }, 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "data": { 08:13:49 DEBUG 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG [...] 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:13:49 DEBUG [...] 08:13:49 DEBUG "result": [ 08:13:49 DEBUG 08:13:49 DEBUG "1.2404917807418367" 08:13:49 DEBUG }, 08:13:49 DEBUG 08:13:49 DEBUG { 08:13:49 DEBUG 08:13:49 DEBUG ] 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG "metric": { 08:13:49 DEBUG 08:13:49 DEBUG } 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG 08:13:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:13:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:13:49 DEBUG ] 08:13:49 DEBUG "14627627008" 08:13:49 DEBUG "instance": "10.142.0.61:9100" 08:13:49 DEBUG }, 08:13:49 DEBUG }, 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG "value": [ 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG ] 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG 08:13:49 DEBUG "0.2" 08:13:49 DEBUG } 08:13:49 DEBUG "6973389.866666667" 08:13:49 DEBUG 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG [...] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 DEBUG 08:13:49 DEBUG ] 08:13:49 DEBUG ] 08:13:49 DEBUG 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 DEBUG 08:13:49 DEBUG } 08:13:49 DEBUG } 08:13:49 DEBUG "pod": "ds-idrepo-1" 08:13:49 INFO [http_cmd]: http status code OK 08:13:49 DEBUG }, 08:13:49 DEBUG --- status code --- 08:13:49 DEBUG "value": [ 08:13:49 DEBUG http status code is 200 (expected 200) 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG --- http response --- 08:13:49 DEBUG "0" 08:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 08:13:49 DEBUG ] 08:13:49 DEBUG ----- output ----- 08:13:49 DEBUG } 08:13:49 DEBUG { 08:13:49 DEBUG ] 08:13:49 DEBUG "status": "success", 08:13:49 DEBUG } 08:13:49 DEBUG "data": { 08:13:49 DEBUG } 08:13:49 DEBUG "resultType": "vector", 08:13:49 DEBUG "result": [ 08:13:49 DEBUG { 08:13:49 DEBUG "metric": { 08:13:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG [...] 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG 08:13:49 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:13:49 DEBUG }, 08:13:49 DEBUG "value": [ 08:13:49 DEBUG 1684566829, 08:13:49 DEBUG "0" 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG ] 08:13:49 DEBUG } 08:13:49 DEBUG } 08:14:24 INFO 08:14:24 INFO [loop_until]: kubectl --namespace=xlou top pods 08:14:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:14:24 INFO [loop_until]: OK (rc = 0) 08:14:24 DEBUG --- stdout --- 08:14:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5524m 9428Mi am-7849cf7bdb-lxfkl 3783m 9366Mi am-7849cf7bdb-td7js 4317m 9424Mi ds-cts-0 7279m 13874Mi ds-cts-1 6838m 13816Mi ds-cts-2 6978m 13877Mi ds-idrepo-0 2046m 13736Mi ds-idrepo-1 12m 13622Mi ds-idrepo-2 15m 13588Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 8m 3395Mi idm-78f4b47cb9-fmzjg 6m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1288m 834Mi 08:14:24 DEBUG --- stderr --- 08:14:24 DEBUG 08:14:36 INFO 08:14:36 INFO [loop_until]: kubectl --namespace=xlou top node 08:14:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:14:36 INFO [loop_until]: OK (rc = 0) 08:14:36 DEBUG --- stdout --- 08:14:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4222m 26% 11578Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 74m 0% 4947Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6146m 38% 11690Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 8022m 50% 12029Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 173m 1% 3096Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1357Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 90m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 61m 0% 14699Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 6896m 43% 15127Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2143m 13% 14835Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 64m 0% 14630Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 7214m 45% 15175Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 6908m 43% 15199Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1375m 8% 2716Mi 4% 08:14:36 DEBUG --- stderr --- 08:14:36 DEBUG 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "ds-idrepo-2" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "ds-cts-2" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "ds-cts-2" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "692.0666666666666" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "406978183.75335515" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566889" 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "instance": "10.142.0.17:9100" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "instance": "10.142.0.61:9100" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "7039739.066666666" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "692.0333333333333" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "ds-idrepo-2" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "230.73333333333332" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "instance": "10.142.0.17:9100" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "instance": "10.142.0.61:9100" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0.0014000000000010005" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "ds-cts-2" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "230.7" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "instance": "10.142.0.17:9100" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "instance": "10.142.0.61:9100" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0.000666666666667363" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "ds-idrepo-2" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG { 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "data": { 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "result": [ 08:14:49 DEBUG { 08:14:49 DEBUG "metric": { 08:14:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG [...] 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG 08:14:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG "0.2" 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684566889" 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684566889" 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566889" 08:14:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566889" 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG { 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG "data": { 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG "result": [ 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG { 08:14:49 DEBUG { 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG "metric": { 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 DEBUG "data": { 08:14:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG "resultType": "vector", 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG }, 08:14:49 DEBUG { 08:14:49 DEBUG "result": [ 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG "value": [ 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG { 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG 08:14:49 DEBUG "data": { 08:14:49 INFO [http_cmd]: http status code OK 08:14:49 DEBUG "metric": { 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG --- status code --- 08:14:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 DEBUG 08:14:49 DEBUG "result": [ 08:14:49 DEBUG http status code is 200 (expected 200) 08:14:49 DEBUG }, 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG [...] 08:14:49 DEBUG { 08:14:49 DEBUG --- http response --- 08:14:49 DEBUG "value": [ 08:14:49 DEBUG { 08:14:49 DEBUG 08:14:49 DEBUG "metric": { 08:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:14:49 DEBUG 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG 08:14:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:14:49 DEBUG ----- output ----- 08:14:49 DEBUG 08:14:49 DEBUG "data": { 08:14:49 DEBUG 08:14:49 DEBUG }, 08:14:49 DEBUG { 08:14:49 DEBUG 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:14:49 DEBUG "value": [ 08:14:49 DEBUG "status": "success", 08:14:49 DEBUG [...] 08:14:49 DEBUG "result": [ 08:14:49 DEBUG }, 08:14:49 DEBUG 08:14:49 DEBUG "data": { 08:14:49 DEBUG 08:14:49 DEBUG { 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG "resultType": "vector", 08:14:49 DEBUG 08:14:49 DEBUG "metric": { 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG 08:14:49 DEBUG "result": [ 08:14:49 DEBUG 08:14:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:14:49 DEBUG "1.299861177298944" 08:14:49 DEBUG [...] 08:14:49 DEBUG { 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG }, 08:14:49 DEBUG ] 08:14:49 DEBUG 08:14:49 DEBUG "metric": { 08:14:49 DEBUG }, 08:14:49 DEBUG "value": [ 08:14:49 DEBUG } 08:14:49 DEBUG 08:14:49 DEBUG "instance": "10.142.0.17:9100" 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG ] 08:14:49 DEBUG 08:14:49 DEBUG }, 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG 08:14:49 DEBUG } 08:14:49 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:14:49 DEBUG "value": [ 08:14:49 DEBUG "0.43322451940515055" 08:14:49 DEBUG 08:14:49 DEBUG } 08:14:49 DEBUG }, 08:14:49 DEBUG 08:14:49 DEBUG ] 08:14:49 DEBUG [...] 08:14:49 DEBUG "value": [ 08:14:49 DEBUG 08:14:49 DEBUG } 08:14:49 DEBUG 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG 08:14:49 DEBUG ] 08:14:49 DEBUG 08:14:49 DEBUG "14628298752" 08:14:49 DEBUG [...] 08:14:49 DEBUG } 08:14:49 DEBUG 08:14:49 DEBUG ] 08:14:49 DEBUG 08:14:49 DEBUG } 08:14:49 DEBUG "pod": "ds-idrepo-1" 08:14:49 DEBUG } 08:14:49 DEBUG 08:14:49 DEBUG }, 08:14:49 DEBUG ] 08:14:49 DEBUG 08:14:49 DEBUG "value": [ 08:14:49 DEBUG } 08:14:49 DEBUG "instance": "10.142.0.61:9100" 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG } 08:14:49 DEBUG }, 08:14:49 DEBUG "0.006135115502865122" 08:14:49 DEBUG "value": [ 08:14:49 DEBUG ] 08:14:49 DEBUG 1684566889, 08:14:49 DEBUG } 08:14:49 DEBUG "7038127.333333333" 08:14:49 DEBUG ] 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 DEBUG } 08:14:49 DEBUG ] 08:14:49 DEBUG } 08:14:49 DEBUG } 08:15:24 INFO 08:15:24 INFO [loop_until]: kubectl --namespace=xlou top pods 08:15:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:15:24 INFO [loop_until]: OK (rc = 0) 08:15:24 DEBUG --- stdout --- 08:15:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5873m 9428Mi am-7849cf7bdb-lxfkl 4331m 9366Mi am-7849cf7bdb-td7js 4907m 9425Mi ds-cts-0 6241m 13756Mi ds-cts-1 7187m 13883Mi ds-cts-2 6659m 13836Mi ds-idrepo-0 2060m 13736Mi ds-idrepo-1 12m 13623Mi ds-idrepo-2 15m 13588Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 8m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 4m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1221m 834Mi 08:15:24 DEBUG --- stderr --- 08:15:24 DEBUG 08:15:36 INFO 08:15:36 INFO [loop_until]: kubectl --namespace=xlou top node 08:15:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:15:36 INFO [loop_until]: OK (rc = 0) 08:15:36 DEBUG --- stdout --- 08:15:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4828m 30% 11576Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 76m 0% 4948Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6139m 38% 11692Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 8369m 52% 12027Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 152m 0% 3108Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 92m 0% 1359Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 91m 0% 5302Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 62m 0% 14702Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 7427m 46% 15204Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2158m 13% 14839Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 64m 0% 14630Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 6677m 42% 15122Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 6357m 40% 15050Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1280m 8% 2713Mi 4% 08:15:36 DEBUG --- stderr --- 08:15:36 DEBUG 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:49 DEBUG { 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG { 08:15:49 DEBUG { 08:15:49 DEBUG "data": { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "data": { 08:15:49 DEBUG "result": [ 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG { 08:15:49 DEBUG "result": [ 08:15:49 DEBUG "result": [ 08:15:49 DEBUG "metric": { 08:15:49 DEBUG { 08:15:49 DEBUG { 08:15:49 DEBUG "pod": "ds-cts-2" 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG }, 08:15:49 DEBUG "pod": "ds-cts-2" 08:15:49 DEBUG "pod": "ds-idrepo-2" 08:15:49 DEBUG "value": [ 08:15:49 DEBUG }, 08:15:49 DEBUG }, 08:15:49 DEBUG 08:15:49 DEBUG "value": [ 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "ds-idrepo-1" 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG }, 08:15:49 DEBUG "pod": "ds-idrepo-1" 08:15:49 DEBUG "pod": "ds-idrepo-1" 08:15:49 DEBUG "value": [ 08:15:49 DEBUG }, 08:15:49 DEBUG }, 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG "value": [ 08:15:49 DEBUG "0" 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG ] 08:15:49 DEBUG "0" 08:15:49 DEBUG "0" 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG ] 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "679.0666666666666" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "ds-idrepo-0" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "9350.378293764676" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "679.1333333333333" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "instance": "10.142.0.17:9100" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "instance": "10.142.0.61:9100" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "6967761.366666666" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "pod": "ds-idrepo-2" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "ds-idrepo-1" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "0" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG { 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG { 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG "data": { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "data": { 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:15:49 DEBUG "result": [ 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "data": { 08:15:49 DEBUG }, 08:15:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "value": [ 08:15:49 DEBUG }, 08:15:49 DEBUG "result": [ 08:15:49 DEBUG 08:15:49 DEBUG "value": [ 08:15:49 DEBUG { 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "metric": { 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "ds-cts-2" 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG }, 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG }, 08:15:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:15:49 DEBUG [...] 08:15:49 DEBUG "value": [ 08:15:49 DEBUG }, 08:15:49 DEBUG 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG "226.46666666666667" 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG 08:15:49 DEBUG ] 08:15:49 DEBUG "226.29999999999998" 08:15:49 DEBUG "pod": "ds-idrepo-1" 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG }, 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG "value": [ 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG "0" 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "instance": "10.142.0.17:9100" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "instance": "10.142.0.61:9100" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "0.0012999999999995756" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "instance": "10.142.0.17:9100" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "instance": "10.142.0.61:9100" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "0" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "pod": "ds-idrepo-2" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "ds-idrepo-1" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "0" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "ds-idrepo-0" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "3.522045395251761" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "0.2" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "instance": "10.142.0.17:9100" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "instance": "10.142.0.61:9100" 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "6964288" 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684566949" 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684566949" 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684566949" 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 INFO [http_cmd]: http status code OK 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- status code --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG http status code is 200 (expected 200) 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:15:49 DEBUG --- http response --- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG ----- output ----- 08:15:49 DEBUG { 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "status": "success", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "data": { 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG "resultType": "vector", 08:15:49 DEBUG "result": [ 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG "result": [ 08:15:49 DEBUG { 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:15:49 DEBUG "metric": { 08:15:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:15:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:15:49 DEBUG }, 08:15:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:15:49 DEBUG }, 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG [...] 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG 08:15:49 DEBUG "pod": "ds-idrepo-0" 08:15:49 DEBUG 08:15:49 DEBUG "pod": "ds-idrepo-0" 08:15:49 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:15:49 DEBUG }, 08:15:49 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:15:49 DEBUG }, 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG }, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "value": [ 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "0.750065223062875" 08:15:49 DEBUG 1684566949, 08:15:49 DEBUG "2.069357477660814" 08:15:49 DEBUG "14628212736" 08:15:49 DEBUG ] 08:15:49 DEBUG "1.2674180406550177" 08:15:49 DEBUG ] 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG ] 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG } 08:15:49 DEBUG } 08:16:24 INFO 08:16:24 INFO [loop_until]: kubectl --namespace=xlou top pods 08:16:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:16:24 INFO [loop_until]: OK (rc = 0) 08:16:24 DEBUG --- stdout --- 08:16:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5951m 9428Mi am-7849cf7bdb-lxfkl 4144m 9367Mi am-7849cf7bdb-td7js 4805m 9425Mi ds-cts-0 6267m 13851Mi ds-cts-1 6717m 13852Mi ds-cts-2 6533m 13827Mi ds-idrepo-0 2086m 13736Mi ds-idrepo-1 13m 13623Mi ds-idrepo-2 14m 13588Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1225m 835Mi 08:16:24 DEBUG --- stderr --- 08:16:24 DEBUG 08:16:36 INFO 08:16:36 INFO [loop_until]: kubectl --namespace=xlou top node 08:16:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:16:36 INFO [loop_until]: OK (rc = 0) 08:16:36 DEBUG --- stdout --- 08:16:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4609m 29% 11578Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4951Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6614m 41% 11698Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 8156m 51% 12032Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 151m 0% 3115Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 84m 0% 1360Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 88m 0% 5302Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 62m 0% 14702Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 7555m 47% 15149Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2150m 13% 14837Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 64m 0% 14632Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 6697m 42% 15147Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 6793m 42% 15212Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1321m 8% 2715Mi 4% 08:16:36 DEBUG --- stderr --- 08:16:36 DEBUG 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "666.5" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "ds-cts-2" 08:16:49 DEBUG "pod": "ds-cts-2" 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "ds-idrepo-1" 08:16:49 DEBUG "pod": "ds-idrepo-1" 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0" 08:16:49 DEBUG "0" 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "ds-idrepo-2" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "ds-idrepo-1" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "am-7849cf7bdb-td7js" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "17299.261783077796" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "666.5333333333333" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567009" 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "instance": "10.142.0.17:9100" 08:16:49 DEBUG "instance": "10.142.0.17:9100" 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "instance": "10.142.0.61:9100" 08:16:49 DEBUG "instance": "10.142.0.61:9100" 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0.0012666666666679778" 08:16:49 DEBUG "6987251.8" 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "ds-idrepo-2" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "ds-idrepo-1" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:16:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:16:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "222.13333333333333" 08:16:49 DEBUG "222.13333333333333" 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "ds-cts-2" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "ds-idrepo-1" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "instance": "10.142.0.17:9100" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "instance": "10.142.0.61:9100" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0.0006666666666665932" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "ds-idrepo-2" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "ds-idrepo-1" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "am-7849cf7bdb-td7js" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0.2" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "instance": "10.142.0.17:9100" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "instance": "10.142.0.61:9100" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "6983667.133333333" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567009" 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567009" 08:16:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567009" 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "14524383232" 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 INFO [http_cmd]: http status code OK 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG --- status code --- 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG http status code is 200 (expected 200) 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG --- http response --- 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG ----- output ----- 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "status": "success", 08:16:49 DEBUG "data": { 08:16:49 DEBUG "data": { 08:16:49 DEBUG "data": { 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "resultType": "vector", 08:16:49 DEBUG "result": [ 08:16:49 DEBUG "result": [ 08:16:49 DEBUG "result": [ 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "metric": { 08:16:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:16:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:16:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG [...] 08:16:49 DEBUG [...] 08:16:49 DEBUG [...] 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG 08:16:49 DEBUG "pod": "am-7849cf7bdb-td7js" 08:16:49 DEBUG "pod": "am-7849cf7bdb-td7js" 08:16:49 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG }, 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG "value": [ 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG 1684567009, 08:16:49 DEBUG "0" 08:16:49 DEBUG "4.372368571089058" 08:16:49 DEBUG "1.228974841742655" 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG ] 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:16:49 DEBUG } 08:17:24 INFO 08:17:24 INFO [loop_until]: kubectl --namespace=xlou top pods 08:17:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:17:24 INFO [loop_until]: OK (rc = 0) 08:17:24 DEBUG --- stdout --- 08:17:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5666m 9428Mi am-7849cf7bdb-lxfkl 3969m 9367Mi am-7849cf7bdb-td7js 4981m 9425Mi ds-cts-0 7101m 13855Mi ds-cts-1 7423m 13877Mi ds-cts-2 7286m 13841Mi ds-idrepo-0 2079m 13736Mi ds-idrepo-1 13m 13623Mi ds-idrepo-2 15m 13588Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 6m 3297Mi lodemon-58bb5f9bf4-7mmfr 9m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1246m 834Mi 08:17:24 DEBUG --- stderr --- 08:17:24 DEBUG 08:17:36 INFO 08:17:36 INFO [loop_until]: kubectl --namespace=xlou top node 08:17:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:17:36 INFO [loop_until]: OK (rc = 0) 08:17:36 DEBUG --- stdout --- 08:17:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4265m 26% 11578Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 71m 0% 4949Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6228m 39% 11696Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 8446m 53% 12030Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 161m 1% 3089Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 80m 0% 1355Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 86m 0% 5301Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 62m 0% 14700Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 7608m 47% 15198Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2160m 13% 14836Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 66m 0% 14635Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 7362m 46% 15179Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 7105m 44% 15201Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1343m 8% 2713Mi 4% 08:17:36 DEBUG --- stderr --- 08:17:36 DEBUG 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "687.2666666666667" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "ds-cts-2" 08:17:49 DEBUG "pod": "ds-cts-2" 08:17:49 DEBUG }, 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "ds-idrepo-1" 08:17:49 DEBUG "pod": "ds-idrepo-1" 08:17:49 DEBUG }, 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0" 08:17:49 DEBUG "0" 08:17:49 DEBUG ] 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "ds-idrepo-2" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "ds-idrepo-1" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "ds-idrepo-1" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "687.2666666666667" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "instance": "10.142.0.17:9100" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "instance": "10.142.0.61:9100" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "7027327.3" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "instance": "10.142.0.17:9100" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "instance": "10.142.0.61:9100" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0.0015333333333349703" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "ds-idrepo-2" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "ds-idrepo-1" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "229.06666666666666" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "229.03333333333333" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "ds-cts-2" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "ds-idrepo-1" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "instance": "10.142.0.17:9100" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "instance": "10.142.0.61:9100" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0.0006666666666667116" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "ds-idrepo-2" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "ds-idrepo-1" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "ds-idrepo-2" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0.2" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "instance": "10.142.0.17:9100" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "instance": "10.142.0.61:9100" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "7025469.4" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567069" 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "1.2934189015437911" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "14592942080" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567069" 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "ds-idrepo-2" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0.7257420712678714" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:17:49 INFO [http_cmd]: http status code OK 08:17:49 DEBUG --- status code --- 08:17:49 DEBUG http status code is 200 (expected 200) 08:17:49 DEBUG --- http response --- 08:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:17:49 DEBUG ----- output ----- 08:17:49 DEBUG { 08:17:49 DEBUG "status": "success", 08:17:49 DEBUG "data": { 08:17:49 DEBUG "resultType": "vector", 08:17:49 DEBUG "result": [ 08:17:49 DEBUG { 08:17:49 DEBUG "metric": { 08:17:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG [...] 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG 08:17:49 DEBUG "pod": "ds-idrepo-1" 08:17:49 DEBUG }, 08:17:49 DEBUG "value": [ 08:17:49 DEBUG 1684567069, 08:17:49 DEBUG "0" 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG ] 08:17:49 DEBUG } 08:17:49 DEBUG } 08:18:24 INFO 08:18:24 INFO [loop_until]: kubectl --namespace=xlou top pods 08:18:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:18:25 INFO [loop_until]: OK (rc = 0) 08:18:25 DEBUG --- stdout --- 08:18:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5222m 9428Mi am-7849cf7bdb-lxfkl 4209m 9367Mi am-7849cf7bdb-td7js 4351m 9426Mi ds-cts-0 6632m 13830Mi ds-cts-1 7056m 13845Mi ds-cts-2 8223m 13876Mi ds-idrepo-0 2004m 13736Mi ds-idrepo-1 13m 13622Mi ds-idrepo-2 16m 13588Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 6m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1164m 835Mi 08:18:25 DEBUG --- stderr --- 08:18:25 DEBUG 08:18:36 INFO 08:18:36 INFO [loop_until]: kubectl --namespace=xlou top node 08:18:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:18:36 INFO [loop_until]: OK (rc = 0) 08:18:36 DEBUG --- stdout --- 08:18:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4741m 29% 11576Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 73m 0% 4947Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 5965m 37% 11690Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7614m 47% 12036Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 155m 0% 3107Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1359Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 87m 0% 5296Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 59m 0% 14702Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 7511m 47% 15195Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2112m 13% 14836Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 64m 0% 14636Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 7266m 45% 15177Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 6752m 42% 15188Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1236m 7% 2716Mi 4% 08:18:36 DEBUG --- stderr --- 08:18:36 DEBUG 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "ds-cts-2" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "ds-idrepo-1" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "0" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "ds-idrepo-2" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "ds-idrepo-1" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "0" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "690.9666666666667" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "ds-cts-2" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "ds-idrepo-1" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "0" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "ds-cts-0" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "654.1302351579031" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "690.8333333333334" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "instance": "10.142.0.17:9100" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "instance": "10.142.0.61:9100" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "7033174.5" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567129" 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "ds-cts-2" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "ds-idrepo-1" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "0" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "instance": "10.142.0.17:9100" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "instance": "10.142.0.61:9100" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "0.0013666666666665604" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "ds-idrepo-2" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "ds-idrepo-1" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "0" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "230.29999999999998" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "230.2" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "instance": "10.142.0.17:9100" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "instance": "10.142.0.61:9100" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "0.000999999999999801" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "ds-idrepo-2" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "ds-idrepo-1" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "0" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "ds-cts-0" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "263.02551203736977" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "0.2" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "instance": "10.142.0.17:9100" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "instance": "10.142.0.61:9100" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "7029397.1" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "14629646336" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "1.2048452073011218" 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567129" 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 INFO [http_cmd]: http status code OK 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG --- status code --- 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG http status code is 200 (expected 200) 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG --- http response --- 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG ----- output ----- 08:18:49 DEBUG { 08:18:49 DEBUG { 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "status": "success", 08:18:49 DEBUG "data": { 08:18:49 DEBUG "data": { 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "resultType": "vector", 08:18:49 DEBUG "result": [ 08:18:49 DEBUG "result": [ 08:18:49 DEBUG { 08:18:49 DEBUG { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "metric": { 08:18:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:18:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:18:49 DEBUG }, 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG [...] 08:18:49 DEBUG [...] 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG 08:18:49 DEBUG "pod": "ds-cts-0" 08:18:49 DEBUG "pod": "ds-cts-0" 08:18:49 DEBUG }, 08:18:49 DEBUG }, 08:18:49 DEBUG "value": [ 08:18:49 DEBUG "value": [ 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG 1684567129, 08:18:49 DEBUG "7.10068482920137" 08:18:49 DEBUG "33.936199944105084" 08:18:49 DEBUG ] 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 DEBUG ] 08:18:49 DEBUG ] 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 DEBUG } 08:18:49 DEBUG } 08:19:25 INFO 08:19:25 INFO [loop_until]: kubectl --namespace=xlou top pods 08:19:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:19:25 INFO [loop_until]: OK (rc = 0) 08:19:25 DEBUG --- stdout --- 08:19:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5818m 9428Mi am-7849cf7bdb-lxfkl 4155m 9367Mi am-7849cf7bdb-td7js 4087m 9425Mi ds-cts-0 7181m 13884Mi ds-cts-1 6815m 13883Mi ds-cts-2 6610m 13852Mi ds-idrepo-0 2040m 13736Mi ds-idrepo-1 13m 13622Mi ds-idrepo-2 15m 13588Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 6m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1236m 834Mi 08:19:25 DEBUG --- stderr --- 08:19:25 DEBUG 08:19:36 INFO 08:19:36 INFO [loop_until]: kubectl --namespace=xlou top node 08:19:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:19:36 INFO [loop_until]: OK (rc = 0) 08:19:36 DEBUG --- stdout --- 08:19:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4874m 30% 11578Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4951Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6435m 40% 11690Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7741m 48% 12033Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 149m 0% 3125Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1358Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 84m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 61m 0% 14702Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 6954m 43% 15208Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2087m 13% 14836Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 65m 0% 14632Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 6891m 43% 15150Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 6884m 43% 15191Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1266m 7% 2717Mi 4% 08:19:36 DEBUG --- stderr --- 08:19:36 DEBUG 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG { 08:19:49 DEBUG { 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG "data": { 08:19:49 DEBUG "data": { 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG "result": [ 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG { 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "data": { 08:19:49 DEBUG "pod": "ds-cts-2" 08:19:49 DEBUG "pod": "ds-cts-2" 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG }, 08:19:49 DEBUG }, 08:19:49 DEBUG "result": [ 08:19:49 DEBUG "value": [ 08:19:49 DEBUG "value": [ 08:19:49 DEBUG { 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "metric": { 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG }, 08:19:49 DEBUG [...] 08:19:49 DEBUG [...] 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "ds-idrepo-1" 08:19:49 DEBUG "pod": "ds-idrepo-1" 08:19:49 DEBUG [...] 08:19:49 DEBUG }, 08:19:49 DEBUG }, 08:19:49 DEBUG 08:19:49 DEBUG "value": [ 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG 08:19:49 DEBUG "0" 08:19:49 DEBUG "0" 08:19:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:19:49 DEBUG ] 08:19:49 DEBUG ] 08:19:49 DEBUG }, 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 DEBUG "value": [ 08:19:49 DEBUG ] 08:19:49 DEBUG ] 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 DEBUG "676.5666666666666" 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "ds-idrepo-2" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "ds-idrepo-1" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "0" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "676.7" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "ds-idrepo-1" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "0" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "instance": "10.142.0.17:9100" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "instance": "10.142.0.61:9100" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "6998483.933333334" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "instance": "10.142.0.17:9100" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "instance": "10.142.0.61:9100" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "0.0013000000000014703" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "ds-cts-2" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "ds-idrepo-1" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "0" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "ds-idrepo-2" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "ds-idrepo-1" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "0" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "225.5" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "225.53333333333333" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "instance": "10.142.0.17:9100" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "instance": "10.142.0.61:9100" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "0.0006666666666672446" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "ds-idrepo-2" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "ds-idrepo-1" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "0" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567189" 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "instance": "10.142.0.17:9100" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "instance": "10.142.0.61:9100" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "6996940.533333333" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "0" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "0.2" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "14630273024" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "data": { 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "result": [ 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG [...] 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:19:49 DEBUG }, 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG "1.2067278690113854" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567189" 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:19:49 DEBUG ----- output ----- 08:19:49 INFO [http_cmd]: http status code OK 08:19:49 DEBUG { 08:19:49 DEBUG --- status code --- 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG http status code is 200 (expected 200) 08:19:49 DEBUG "data": { 08:19:49 DEBUG --- http response --- 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:19:49 DEBUG "result": [ 08:19:49 DEBUG ----- output ----- 08:19:49 DEBUG { 08:19:49 DEBUG { 08:19:49 DEBUG "metric": { 08:19:49 DEBUG "status": "success", 08:19:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:19:49 DEBUG "data": { 08:19:49 DEBUG }, 08:19:49 DEBUG "resultType": "vector", 08:19:49 DEBUG "value": [ 08:19:49 DEBUG "result": [ 08:19:49 DEBUG 08:19:49 DEBUG { 08:19:49 DEBUG 08:19:49 DEBUG "metric": { 08:19:49 DEBUG 08:19:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:19:49 DEBUG [...] 08:19:49 DEBUG }, 08:19:49 DEBUG 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG 08:19:49 DEBUG "pod": "ds-idrepo-1" 08:19:49 DEBUG 08:19:49 DEBUG }, 08:19:49 DEBUG [...] 08:19:49 DEBUG "value": [ 08:19:49 DEBUG 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG 08:19:49 DEBUG "0" 08:19:49 DEBUG 08:19:49 DEBUG ] 08:19:49 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:19:49 DEBUG } 08:19:49 DEBUG }, 08:19:49 DEBUG ] 08:19:49 DEBUG "value": [ 08:19:49 DEBUG } 08:19:49 DEBUG 1684567189, 08:19:49 DEBUG } 08:19:49 DEBUG "0" 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG ] 08:19:49 DEBUG } 08:19:49 DEBUG } 08:20:25 INFO 08:20:25 INFO [loop_until]: kubectl --namespace=xlou top pods 08:20:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:20:25 INFO [loop_until]: OK (rc = 0) 08:20:25 DEBUG --- stdout --- 08:20:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5849m 9428Mi am-7849cf7bdb-lxfkl 4156m 9367Mi am-7849cf7bdb-td7js 4901m 9426Mi ds-cts-0 6813m 13824Mi ds-cts-1 7640m 13811Mi ds-cts-2 7073m 13876Mi ds-idrepo-0 2160m 13737Mi ds-idrepo-1 12m 13622Mi ds-idrepo-2 15m 13589Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 6m 3297Mi lodemon-58bb5f9bf4-7mmfr 8m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1230m 835Mi 08:20:25 DEBUG --- stderr --- 08:20:25 DEBUG 08:20:36 INFO 08:20:36 INFO [loop_until]: kubectl --namespace=xlou top node 08:20:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:20:37 INFO [loop_until]: OK (rc = 0) 08:20:37 DEBUG --- stdout --- 08:20:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4931m 31% 11578Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 69m 0% 4951Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6584m 41% 11691Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 8131m 51% 12033Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 153m 0% 3126Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 79m 0% 1352Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 81m 0% 5302Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 62m 0% 14699Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 7139m 44% 15157Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2178m 13% 14836Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 65m 0% 14633Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 7437m 46% 15188Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 7130m 44% 15178Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1294m 8% 2715Mi 4% 08:20:37 DEBUG --- stderr --- 08:20:37 DEBUG 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "693.2" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG { 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:20:49 DEBUG "data": { 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "pod": "ds-cts-2" 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG }, 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "value": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "ds-cts-2" 08:20:49 DEBUG 08:20:49 DEBUG "pod": "ds-idrepo-2" 08:20:49 DEBUG }, 08:20:49 DEBUG 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG [...] 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG "pod": "ds-idrepo-1" 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG }, 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG 08:20:49 DEBUG "pod": "ds-idrepo-1" 08:20:49 DEBUG "0" 08:20:49 DEBUG "pod": "ds-idrepo-1" 08:20:49 DEBUG }, 08:20:49 DEBUG ] 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG } 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG ] 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "0" 08:20:49 DEBUG } 08:20:49 DEBUG "0" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "292173399.8640999" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "693.2333333333333" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "instance": "10.142.0.17:9100" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "instance": "10.142.0.61:9100" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "6973332.533333333" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567249" 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "instance": "10.142.0.17:9100" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "instance": "10.142.0.61:9100" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "0.0013333333333321205" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "ds-cts-2" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "ds-idrepo-1" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "0" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "ds-idrepo-2" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "ds-idrepo-1" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "0" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "230.9" 08:20:49 DEBUG ] 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "231.23333333333332" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "ds-idrepo-2" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "ds-idrepo-1" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "0" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567249" 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "instance": "10.142.0.17:9100" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "instance": "10.142.0.61:9100" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "0.00033333333333350387" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "instance": "10.142.0.17:9100" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "instance": "10.142.0.61:9100" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "6969669.7" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "0" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "0.2" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "14625492992" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567249" 08:20:49 INFO [http_cmd]: http status code OK 08:20:49 DEBUG --- status code --- 08:20:49 DEBUG http status code is 200 (expected 200) 08:20:49 DEBUG --- http response --- 08:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:20:49 DEBUG ----- output ----- 08:20:49 DEBUG { 08:20:49 DEBUG "status": "success", 08:20:49 DEBUG "data": { 08:20:49 DEBUG "resultType": "vector", 08:20:49 DEBUG "result": [ 08:20:49 DEBUG { 08:20:49 DEBUG "metric": { 08:20:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG [...] 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG 08:20:49 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:20:49 DEBUG }, 08:20:49 DEBUG "value": [ 08:20:49 DEBUG 1684567249, 08:20:49 DEBUG "1.2187731367498507" 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG ] 08:20:49 DEBUG } 08:20:49 DEBUG } 08:20:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567250" 08:20:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567250" 08:20:50 INFO [http_cmd]: http status code OK 08:20:50 INFO [http_cmd]: http status code OK 08:20:50 DEBUG --- status code --- 08:20:50 DEBUG --- status code --- 08:20:50 DEBUG http status code is 200 (expected 200) 08:20:50 DEBUG http status code is 200 (expected 200) 08:20:50 DEBUG --- http response --- 08:20:50 DEBUG --- http response --- 08:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:20:50 DEBUG ----- output ----- 08:20:50 DEBUG ----- output ----- 08:20:50 DEBUG { 08:20:50 DEBUG { 08:20:50 DEBUG "status": "success", 08:20:50 DEBUG "status": "success", 08:20:50 DEBUG "data": { 08:20:50 DEBUG "data": { 08:20:50 DEBUG "resultType": "vector", 08:20:50 DEBUG "resultType": "vector", 08:20:50 DEBUG "result": [ 08:20:50 DEBUG "result": [ 08:20:50 DEBUG { 08:20:50 DEBUG { 08:20:50 DEBUG "metric": { 08:20:50 DEBUG "metric": { 08:20:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:20:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:20:50 DEBUG }, 08:20:50 DEBUG }, 08:20:50 DEBUG "value": [ 08:20:50 DEBUG "value": [ 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG [...] 08:20:50 DEBUG [...] 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG 08:20:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:20:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:20:50 DEBUG }, 08:20:50 DEBUG }, 08:20:50 DEBUG "value": [ 08:20:50 DEBUG "value": [ 08:20:50 DEBUG 1684567250, 08:20:50 DEBUG 1684567250, 08:20:50 DEBUG "2.480299943098627" 08:20:50 DEBUG "0" 08:20:50 DEBUG ] 08:20:50 DEBUG ] 08:20:50 DEBUG } 08:20:50 DEBUG } 08:20:50 DEBUG ] 08:20:50 DEBUG ] 08:20:50 DEBUG } 08:20:50 DEBUG } 08:20:50 DEBUG } 08:20:50 DEBUG } 08:21:25 INFO 08:21:25 INFO [loop_until]: kubectl --namespace=xlou top pods 08:21:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:21:25 INFO [loop_until]: OK (rc = 0) 08:21:25 DEBUG --- stdout --- 08:21:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5845m 9428Mi am-7849cf7bdb-lxfkl 4265m 9368Mi am-7849cf7bdb-td7js 4855m 9426Mi ds-cts-0 7137m 13884Mi ds-cts-1 6858m 13805Mi ds-cts-2 7100m 13876Mi ds-idrepo-0 2150m 13736Mi ds-idrepo-1 12m 13623Mi ds-idrepo-2 14m 13589Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 6m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1237m 835Mi 08:21:25 DEBUG --- stderr --- 08:21:25 DEBUG 08:21:37 INFO 08:21:37 INFO [loop_until]: kubectl --namespace=xlou top node 08:21:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:21:37 INFO [loop_until]: OK (rc = 0) 08:21:37 DEBUG --- stdout --- 08:21:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4791m 30% 11579Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 70m 0% 4947Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6323m 39% 11690Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 8347m 52% 12029Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 151m 0% 3144Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1353Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 82m 0% 5296Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14714Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 7348m 46% 15143Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2172m 13% 14838Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 62m 0% 14634Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 7453m 46% 15182Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 7655m 48% 15121Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1282m 8% 2715Mi 4% 08:21:37 DEBUG --- stderr --- 08:21:37 DEBUG 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "ds-idrepo-2" 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:21:49 DEBUG }, 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG { 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG "data": { 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "result": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG { 08:21:49 DEBUG 08:21:49 DEBUG "pod": "ds-idrepo-1" 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:21:49 DEBUG }, 08:21:49 DEBUG "pod": "ds-cts-2" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "0" 08:21:49 DEBUG 08:21:49 DEBUG "671.8" 08:21:49 DEBUG ] 08:21:49 DEBUG 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG [...] 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 DEBUG 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG } 08:21:49 DEBUG 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG "pod": "ds-idrepo-1" 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG }, 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG "value": [ 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG "0" 08:21:49 DEBUG { 08:21:49 DEBUG ] 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG } 08:21:49 DEBUG "data": { 08:21:49 DEBUG ] 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG } 08:21:49 DEBUG "result": [ 08:21:49 DEBUG } 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "ds-cts-2" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "ds-idrepo-1" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "0" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "am-7849cf7bdb-td7js" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "0" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "671.7666666666667" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567309" 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "ds-idrepo-2" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "ds-idrepo-1" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "0" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "instance": "10.142.0.17:9100" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "instance": "10.142.0.61:9100" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "6966163.7" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567309" 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "ds-cts-2" 08:21:49 DEBUG "instance": "10.142.0.17:9100" 08:21:49 DEBUG }, 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "ds-idrepo-1" 08:21:49 DEBUG "instance": "10.142.0.61:9100" 08:21:49 DEBUG }, 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "0" 08:21:49 DEBUG "0.0015666666666675154" 08:21:49 DEBUG ] 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "ds-idrepo-2" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "ds-idrepo-1" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "0" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "223.9" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "223.96666666666667" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567309" 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567309" 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "instance": "10.142.0.17:9100" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "instance": "10.142.0.61:9100" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "0.0003333333333330302" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "instance": "10.142.0.17:9100" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "instance": "10.142.0.61:9100" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "6962676.433333334" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "0" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567309" 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "0.2" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567309" 08:21:49 INFO [http_cmd]: http status code OK 08:21:49 DEBUG --- status code --- 08:21:49 DEBUG http status code is 200 (expected 200) 08:21:49 DEBUG --- http response --- 08:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:49 DEBUG ----- output ----- 08:21:49 DEBUG { 08:21:49 DEBUG "status": "success", 08:21:49 DEBUG "data": { 08:21:49 DEBUG "resultType": "vector", 08:21:49 DEBUG "result": [ 08:21:49 DEBUG { 08:21:49 DEBUG "metric": { 08:21:49 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG [...] 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG 08:21:49 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:21:49 DEBUG }, 08:21:49 DEBUG "value": [ 08:21:49 DEBUG 1684567309, 08:21:49 DEBUG "14630248448" 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG ] 08:21:49 DEBUG } 08:21:49 DEBUG } 08:21:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567310" 08:21:50 INFO [http_cmd]: http status code OK 08:21:50 DEBUG --- status code --- 08:21:50 DEBUG http status code is 200 (expected 200) 08:21:50 DEBUG --- http response --- 08:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:21:50 DEBUG ----- output ----- 08:21:50 DEBUG { 08:21:50 DEBUG "status": "success", 08:21:50 DEBUG "data": { 08:21:50 DEBUG "resultType": "vector", 08:21:50 DEBUG "result": [ 08:21:50 DEBUG { 08:21:50 DEBUG "metric": { 08:21:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:21:50 DEBUG }, 08:21:50 DEBUG "value": [ 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG [...] 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:21:50 DEBUG }, 08:21:50 DEBUG "value": [ 08:21:50 DEBUG 1684567310, 08:21:50 DEBUG "1.236937141668248" 08:21:50 DEBUG ] 08:21:50 DEBUG } 08:21:50 DEBUG ] 08:21:50 DEBUG } 08:21:50 DEBUG } 08:21:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567310" 08:21:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567310" 08:21:50 INFO [http_cmd]: http status code OK 08:21:50 INFO [http_cmd]: http status code OK 08:21:50 DEBUG --- status code --- 08:21:50 DEBUG --- status code --- 08:21:50 DEBUG http status code is 200 (expected 200) 08:21:50 DEBUG http status code is 200 (expected 200) 08:21:50 DEBUG --- http response --- 08:21:50 DEBUG --- http response --- 08:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:21:50 DEBUG ----- output ----- 08:21:50 DEBUG ----- output ----- 08:21:50 DEBUG { 08:21:50 DEBUG { 08:21:50 DEBUG "status": "success", 08:21:50 DEBUG "status": "success", 08:21:50 DEBUG "data": { 08:21:50 DEBUG "data": { 08:21:50 DEBUG "resultType": "vector", 08:21:50 DEBUG "resultType": "vector", 08:21:50 DEBUG "result": [ 08:21:50 DEBUG "result": [ 08:21:50 DEBUG { 08:21:50 DEBUG { 08:21:50 DEBUG "metric": { 08:21:50 DEBUG "metric": { 08:21:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:21:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:21:50 DEBUG }, 08:21:50 DEBUG }, 08:21:50 DEBUG "value": [ 08:21:50 DEBUG "value": [ 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG [...] 08:21:50 DEBUG [...] 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG 08:21:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:21:50 DEBUG "pod": "am-7849cf7bdb-td7js" 08:21:50 DEBUG }, 08:21:50 DEBUG }, 08:21:50 DEBUG "value": [ 08:21:50 DEBUG "value": [ 08:21:50 DEBUG 1684567310, 08:21:50 DEBUG 1684567310, 08:21:50 DEBUG "0" 08:21:50 DEBUG "0" 08:21:50 DEBUG ] 08:21:50 DEBUG ] 08:21:50 DEBUG } 08:21:50 DEBUG } 08:21:50 DEBUG ] 08:21:50 DEBUG ] 08:21:50 DEBUG } 08:21:50 DEBUG } 08:21:50 DEBUG } 08:21:50 DEBUG } 08:22:25 INFO 08:22:25 INFO [loop_until]: kubectl --namespace=xlou top pods 08:22:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:22:25 INFO [loop_until]: OK (rc = 0) 08:22:25 DEBUG --- stdout --- 08:22:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5882m 9443Mi am-7849cf7bdb-lxfkl 3978m 9368Mi am-7849cf7bdb-td7js 4439m 9426Mi ds-cts-0 6722m 13879Mi ds-cts-1 7627m 13880Mi ds-cts-2 6510m 13778Mi ds-idrepo-0 2071m 13736Mi ds-idrepo-1 12m 13623Mi ds-idrepo-2 15m 13589Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1273m 835Mi 08:22:25 DEBUG --- stderr --- 08:22:25 DEBUG 08:22:37 INFO 08:22:37 INFO [loop_until]: kubectl --namespace=xlou top node 08:22:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:22:37 INFO [loop_until]: OK (rc = 0) 08:22:37 DEBUG --- stdout --- 08:22:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4419m 27% 11577Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 4947Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6275m 39% 11708Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 8141m 51% 12028Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 148m 0% 3139Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1354Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 80m 0% 5300Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 59m 0% 14714Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 7534m 47% 15198Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2157m 13% 14837Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 66m 0% 14634Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 6570m 41% 15181Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 6826m 42% 15204Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1350m 8% 2716Mi 4% 08:22:37 DEBUG --- stderr --- 08:22:37 DEBUG 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG { 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG { 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "data": { 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "result": [ 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG "result": [ 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG { 08:22:49 DEBUG { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "pod": "ds-cts-2" 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:22:49 DEBUG }, 08:22:49 DEBUG "pod": "ds-cts-2" 08:22:49 DEBUG "pod": "ds-idrepo-2" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG }, 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG "value": [ 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG [...] 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "pod": "ds-idrepo-1" 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:22:49 DEBUG }, 08:22:49 DEBUG "pod": "ds-idrepo-1" 08:22:49 DEBUG "pod": "ds-idrepo-1" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG }, 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "0" 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "682.8333333333334" 08:22:49 DEBUG ] 08:22:49 DEBUG "0" 08:22:49 DEBUG "0" 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "pod": "ds-idrepo-1" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "11737.650893720673" 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "682.7333333333333" 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567369" 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "pod": "ds-idrepo-2" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "pod": "ds-idrepo-1" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "0" 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "pod": "ds-idrepo-2" 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG }, 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG "value": [ 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:49 DEBUG 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG [...] 08:22:49 DEBUG { 08:22:49 DEBUG 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG 08:22:49 DEBUG "data": { 08:22:49 DEBUG 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "pod": "ds-idrepo-1" 08:22:49 DEBUG "result": [ 08:22:49 DEBUG }, 08:22:49 DEBUG { 08:22:49 DEBUG "value": [ 08:22:49 DEBUG "metric": { 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "instance": "10.142.0.17:9100" 08:22:49 DEBUG "0" 08:22:49 DEBUG }, 08:22:49 DEBUG ] 08:22:49 DEBUG "value": [ 08:22:49 DEBUG } 08:22:49 DEBUG 08:22:49 DEBUG ] 08:22:49 DEBUG 08:22:49 DEBUG } 08:22:49 DEBUG 08:22:49 DEBUG } 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "instance": "10.142.0.61:9100" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "6963249.866666667" 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567369" 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG { 08:22:49 DEBUG { 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "data": { 08:22:49 DEBUG "data": { 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "result": [ 08:22:49 DEBUG "result": [ 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG { 08:22:49 DEBUG { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:22:49 DEBUG "pod": "ds-cts-2" 08:22:49 DEBUG "instance": "10.142.0.17:9100" 08:22:49 DEBUG }, 08:22:49 DEBUG }, 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG "value": [ 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG [...] 08:22:49 DEBUG [...] 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:22:49 DEBUG "pod": "ds-idrepo-1" 08:22:49 DEBUG "instance": "10.142.0.61:9100" 08:22:49 DEBUG }, 08:22:49 DEBUG }, 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG "value": [ 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "227.63333333333333" 08:22:49 DEBUG "0" 08:22:49 DEBUG "0.0015000000000014778" 08:22:49 DEBUG ] 08:22:49 DEBUG ] 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG ] 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567369" 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG { 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG "data": { 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "data": { 08:22:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG }, 08:22:49 DEBUG "result": [ 08:22:49 DEBUG "value": [ 08:22:49 DEBUG { 08:22:49 DEBUG 08:22:49 DEBUG "metric": { 08:22:49 DEBUG 08:22:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:22:49 DEBUG 08:22:49 DEBUG }, 08:22:49 DEBUG [...] 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "pod": "ds-idrepo-1" 08:22:49 DEBUG [...] 08:22:49 DEBUG }, 08:22:49 DEBUG 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG 08:22:49 DEBUG "0" 08:22:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:22:49 DEBUG ] 08:22:49 DEBUG }, 08:22:49 DEBUG } 08:22:49 DEBUG "value": [ 08:22:49 DEBUG ] 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG } 08:22:49 DEBUG "227.56666666666666" 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567369" 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "instance": "10.142.0.17:9100" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "instance": "10.142.0.61:9100" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "6961693.1" 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567369" 08:22:49 INFO [http_cmd]: http status code OK 08:22:49 DEBUG --- status code --- 08:22:49 DEBUG http status code is 200 (expected 200) 08:22:49 DEBUG --- http response --- 08:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:49 DEBUG ----- output ----- 08:22:49 DEBUG { 08:22:49 DEBUG "status": "success", 08:22:49 DEBUG "data": { 08:22:49 DEBUG "resultType": "vector", 08:22:49 DEBUG "result": [ 08:22:49 DEBUG { 08:22:49 DEBUG "metric": { 08:22:49 DEBUG "instance": "10.142.0.17:9100" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG [...] 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG 08:22:49 DEBUG "instance": "10.142.0.61:9100" 08:22:49 DEBUG }, 08:22:49 DEBUG "value": [ 08:22:49 DEBUG 1684567369, 08:22:49 DEBUG "0.000333333333333267" 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG ] 08:22:49 DEBUG } 08:22:49 DEBUG } 08:22:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567370" 08:22:50 INFO [http_cmd]: http status code OK 08:22:50 DEBUG --- status code --- 08:22:50 DEBUG http status code is 200 (expected 200) 08:22:50 DEBUG --- http response --- 08:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:22:50 DEBUG ----- output ----- 08:22:50 DEBUG { 08:22:50 DEBUG "status": "success", 08:22:50 DEBUG "data": { 08:22:50 DEBUG "resultType": "vector", 08:22:50 DEBUG "result": [ 08:22:50 DEBUG { 08:22:50 DEBUG "metric": { 08:22:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG [...] 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 1684567370, 08:22:50 DEBUG "0.2" 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG } 08:22:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567370" 08:22:50 INFO [http_cmd]: http status code OK 08:22:50 DEBUG --- status code --- 08:22:50 DEBUG http status code is 200 (expected 200) 08:22:50 DEBUG --- http response --- 08:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:50 DEBUG ----- output ----- 08:22:50 DEBUG { 08:22:50 DEBUG "status": "success", 08:22:50 DEBUG "data": { 08:22:50 DEBUG "resultType": "vector", 08:22:50 DEBUG "result": [ 08:22:50 DEBUG { 08:22:50 DEBUG "metric": { 08:22:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG [...] 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 1684567370, 08:22:50 DEBUG "14527426560" 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG } 08:22:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567370" 08:22:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567370" 08:22:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567370" 08:22:50 INFO [http_cmd]: http status code OK 08:22:50 DEBUG --- status code --- 08:22:50 DEBUG http status code is 200 (expected 200) 08:22:50 DEBUG --- http response --- 08:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:22:50 DEBUG ----- output ----- 08:22:50 DEBUG { 08:22:50 DEBUG "status": "success", 08:22:50 DEBUG "data": { 08:22:50 DEBUG "resultType": "vector", 08:22:50 DEBUG "result": [ 08:22:50 DEBUG { 08:22:50 DEBUG "metric": { 08:22:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG [...] 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 1684567370, 08:22:50 DEBUG "1.2681101535643864" 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG } 08:22:50 INFO [http_cmd]: http status code OK 08:22:50 DEBUG --- status code --- 08:22:50 DEBUG http status code is 200 (expected 200) 08:22:50 DEBUG --- http response --- 08:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:22:50 DEBUG ----- output ----- 08:22:50 DEBUG { 08:22:50 DEBUG "status": "success", 08:22:50 DEBUG "data": { 08:22:50 DEBUG "resultType": "vector", 08:22:50 DEBUG "result": [ 08:22:50 DEBUG { 08:22:50 DEBUG "metric": { 08:22:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG [...] 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG "pod": "ds-idrepo-1" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 1684567370, 08:22:50 DEBUG "0.9313321631980513" 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG } 08:22:50 INFO [http_cmd]: http status code OK 08:22:50 DEBUG --- status code --- 08:22:50 DEBUG http status code is 200 (expected 200) 08:22:50 DEBUG --- http response --- 08:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:22:50 DEBUG ----- output ----- 08:22:50 DEBUG { 08:22:50 DEBUG "status": "success", 08:22:50 DEBUG "data": { 08:22:50 DEBUG "resultType": "vector", 08:22:50 DEBUG "result": [ 08:22:50 DEBUG { 08:22:50 DEBUG "metric": { 08:22:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG [...] 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG 08:22:50 DEBUG "pod": "ds-idrepo-1" 08:22:50 DEBUG }, 08:22:50 DEBUG "value": [ 08:22:50 DEBUG 1684567370, 08:22:50 DEBUG "0.011985357094269745" 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG ] 08:22:50 DEBUG } 08:22:50 DEBUG } 08:23:25 INFO 08:23:25 INFO [loop_until]: kubectl --namespace=xlou top pods 08:23:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:23:25 INFO [loop_until]: OK (rc = 0) 08:23:25 DEBUG --- stdout --- 08:23:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 6016m 9428Mi am-7849cf7bdb-lxfkl 3809m 9368Mi am-7849cf7bdb-td7js 4267m 9426Mi ds-cts-0 7106m 13823Mi ds-cts-1 7176m 13800Mi ds-cts-2 6722m 13877Mi ds-idrepo-0 2072m 13736Mi ds-idrepo-1 12m 13622Mi ds-idrepo-2 17m 13589Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1193m 835Mi 08:23:25 DEBUG --- stderr --- 08:23:25 DEBUG 08:23:37 INFO 08:23:37 INFO [loop_until]: kubectl --namespace=xlou top node 08:23:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:23:37 INFO [loop_until]: OK (rc = 0) 08:23:37 DEBUG --- stdout --- 08:23:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4839m 30% 11579Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 73m 0% 4950Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6419m 40% 11694Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7723m 48% 12028Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 161m 1% 3100Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1354Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 78m 0% 5300Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 62m 0% 14704Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 6887m 43% 15169Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2163m 13% 14838Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14634Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 7012m 44% 15182Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 7127m 44% 15174Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1275m 8% 2715Mi 4% 08:23:37 DEBUG --- stderr --- 08:23:37 DEBUG 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "pod": "ds-cts-2" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "ds-idrepo-1" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "0" 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "pod": "ds-idrepo-2" 08:23:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:23:49 DEBUG }, 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "ds-idrepo-1" 08:23:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:23:49 DEBUG }, 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "0" 08:23:49 DEBUG "670.9333333333333" 08:23:49 DEBUG ] 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "pod": "ds-cts-2" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "ds-idrepo-1" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "0" 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "ds-idrepo-2" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "0" 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "670.8333333333334" 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567429" 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "pod": "ds-idrepo-2" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "ds-idrepo-1" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "0" 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "pod": "ds-cts-2" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "ds-idrepo-1" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "0" 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG { 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG "data": { 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:49 DEBUG "result": [ 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "pod": "ds-idrepo-2" 08:23:49 DEBUG "data": { 08:23:49 DEBUG }, 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "value": [ 08:23:49 DEBUG "result": [ 08:23:49 DEBUG 08:23:49 DEBUG { 08:23:49 DEBUG 08:23:49 DEBUG "metric": { 08:23:49 DEBUG 08:23:49 DEBUG "instance": "10.142.0.17:9100" 08:23:49 DEBUG [...] 08:23:49 DEBUG }, 08:23:49 DEBUG 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "ds-idrepo-1" 08:23:49 DEBUG 08:23:49 DEBUG }, 08:23:49 DEBUG [...] 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG 08:23:49 DEBUG "0" 08:23:49 DEBUG 08:23:49 DEBUG ] 08:23:49 DEBUG "instance": "10.142.0.61:9100" 08:23:49 DEBUG } 08:23:49 DEBUG }, 08:23:49 DEBUG ] 08:23:49 DEBUG "value": [ 08:23:49 DEBUG } 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG } 08:23:49 DEBUG "6985306.866666667" 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567429" 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567429" 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:49 DEBUG { 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG { 08:23:49 DEBUG "data": { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "result": [ 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG { 08:23:49 DEBUG "result": [ 08:23:49 DEBUG "metric": { 08:23:49 DEBUG { 08:23:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:23:49 DEBUG "metric": { 08:23:49 DEBUG }, 08:23:49 DEBUG "instance": "10.142.0.17:9100" 08:23:49 DEBUG "value": [ 08:23:49 DEBUG }, 08:23:49 DEBUG 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:23:49 DEBUG 08:23:49 DEBUG }, 08:23:49 DEBUG "instance": "10.142.0.61:9100" 08:23:49 DEBUG "value": [ 08:23:49 DEBUG }, 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG "223.63333333333333" 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG ] 08:23:49 DEBUG "0.0015666666666656207" 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567429" 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "instance": "10.142.0.17:9100" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "instance": "10.142.0.61:9100" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "0.000999999999999801" 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:23:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:23:49 DEBUG }, 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:23:49 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:23:49 DEBUG }, 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "223.66666666666666" 08:23:49 DEBUG "0" 08:23:49 DEBUG ] 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567429" 08:23:49 INFO [http_cmd]: http status code OK 08:23:49 DEBUG --- status code --- 08:23:49 DEBUG http status code is 200 (expected 200) 08:23:49 DEBUG --- http response --- 08:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:49 DEBUG ----- output ----- 08:23:49 DEBUG { 08:23:49 DEBUG "status": "success", 08:23:49 DEBUG "data": { 08:23:49 DEBUG "resultType": "vector", 08:23:49 DEBUG "result": [ 08:23:49 DEBUG { 08:23:49 DEBUG "metric": { 08:23:49 DEBUG "instance": "10.142.0.17:9100" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG [...] 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG 08:23:49 DEBUG "instance": "10.142.0.61:9100" 08:23:49 DEBUG }, 08:23:49 DEBUG "value": [ 08:23:49 DEBUG 1684567429, 08:23:49 DEBUG "6981727.433333334" 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG ] 08:23:49 DEBUG } 08:23:49 DEBUG } 08:23:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567430" 08:23:50 INFO [http_cmd]: http status code OK 08:23:50 DEBUG --- status code --- 08:23:50 DEBUG http status code is 200 (expected 200) 08:23:50 DEBUG --- http response --- 08:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:23:50 DEBUG ----- output ----- 08:23:50 DEBUG { 08:23:50 DEBUG "status": "success", 08:23:50 DEBUG "data": { 08:23:50 DEBUG "resultType": "vector", 08:23:50 DEBUG "result": [ 08:23:50 DEBUG { 08:23:50 DEBUG "metric": { 08:23:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:23:50 DEBUG }, 08:23:50 DEBUG "value": [ 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG [...] 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:23:50 DEBUG }, 08:23:50 DEBUG "value": [ 08:23:50 DEBUG 1684567430, 08:23:50 DEBUG "0.2" 08:23:50 DEBUG ] 08:23:50 DEBUG } 08:23:50 DEBUG ] 08:23:50 DEBUG } 08:23:50 DEBUG } 08:23:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567430" 08:23:50 INFO [http_cmd]: http status code OK 08:23:50 DEBUG --- status code --- 08:23:50 DEBUG http status code is 200 (expected 200) 08:23:50 DEBUG --- http response --- 08:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:50 DEBUG ----- output ----- 08:23:50 DEBUG { 08:23:50 DEBUG "status": "success", 08:23:50 DEBUG "data": { 08:23:50 DEBUG "resultType": "vector", 08:23:50 DEBUG "result": [ 08:23:50 DEBUG { 08:23:50 DEBUG "metric": { 08:23:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:23:50 DEBUG }, 08:23:50 DEBUG "value": [ 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG [...] 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:23:50 DEBUG }, 08:23:50 DEBUG "value": [ 08:23:50 DEBUG 1684567430, 08:23:50 DEBUG "14631915520" 08:23:50 DEBUG ] 08:23:50 DEBUG } 08:23:50 DEBUG ] 08:23:50 DEBUG } 08:23:50 DEBUG } 08:23:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567430" 08:23:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567430" 08:23:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567430" 08:23:50 INFO [http_cmd]: http status code OK 08:23:50 INFO [http_cmd]: http status code OK 08:23:50 DEBUG --- status code --- 08:23:50 DEBUG --- status code --- 08:23:50 DEBUG http status code is 200 (expected 200) 08:23:50 DEBUG http status code is 200 (expected 200) 08:23:50 DEBUG --- http response --- 08:23:50 DEBUG --- http response --- 08:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:23:50 DEBUG ----- output ----- 08:23:50 DEBUG ----- output ----- 08:23:50 DEBUG { 08:23:50 DEBUG { 08:23:50 DEBUG "status": "success", 08:23:50 DEBUG "status": "success", 08:23:50 DEBUG "data": { 08:23:50 DEBUG "data": { 08:23:50 DEBUG "resultType": "vector", 08:23:50 DEBUG "resultType": "vector", 08:23:50 DEBUG "result": [ 08:23:50 DEBUG "result": [ 08:23:50 DEBUG { 08:23:50 DEBUG { 08:23:50 DEBUG "metric": { 08:23:50 DEBUG "metric": { 08:23:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:23:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:23:50 DEBUG }, 08:23:50 DEBUG }, 08:23:50 DEBUG "value": [ 08:23:50 DEBUG "value": [ 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG [...] 08:23:50 DEBUG [...] 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG "pod": "ds-cts-0" 08:23:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:23:50 DEBUG }, 08:23:50 DEBUG }, 08:23:50 DEBUG "value": [ 08:23:50 DEBUG "value": [ 08:23:50 DEBUG 1684567430, 08:23:50 DEBUG 1684567430, 08:23:50 DEBUG "24.614408313041128" 08:23:50 DEBUG "1.2349978134533206" 08:23:50 DEBUG ] 08:23:50 DEBUG ] 08:23:50 DEBUG } 08:23:50 DEBUG } 08:23:50 DEBUG ] 08:23:50 DEBUG ] 08:23:50 DEBUG } 08:23:50 DEBUG } 08:23:50 DEBUG } 08:23:50 DEBUG } 08:23:50 INFO [http_cmd]: http status code OK 08:23:50 DEBUG --- status code --- 08:23:50 DEBUG http status code is 200 (expected 200) 08:23:50 DEBUG --- http response --- 08:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:23:50 DEBUG ----- output ----- 08:23:50 DEBUG { 08:23:50 DEBUG "status": "success", 08:23:50 DEBUG "data": { 08:23:50 DEBUG "resultType": "vector", 08:23:50 DEBUG "result": [ 08:23:50 DEBUG { 08:23:50 DEBUG "metric": { 08:23:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:23:50 DEBUG }, 08:23:50 DEBUG "value": [ 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG [...] 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG 08:23:50 DEBUG "pod": "ds-cts-0" 08:23:50 DEBUG }, 08:23:50 DEBUG "value": [ 08:23:50 DEBUG 1684567430, 08:23:50 DEBUG "4.480330964689675" 08:23:50 DEBUG ] 08:23:50 DEBUG } 08:23:50 DEBUG ] 08:23:50 DEBUG } 08:23:50 DEBUG } 08:24:25 INFO 08:24:25 INFO [loop_until]: kubectl --namespace=xlou top pods 08:24:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:24:25 INFO [loop_until]: OK (rc = 0) 08:24:25 DEBUG --- stdout --- 08:24:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5446m 9429Mi am-7849cf7bdb-lxfkl 4273m 9368Mi am-7849cf7bdb-td7js 4290m 9426Mi ds-cts-0 7072m 13882Mi ds-cts-1 6990m 13880Mi ds-cts-2 6641m 13835Mi ds-idrepo-0 2031m 13736Mi ds-idrepo-1 11m 13622Mi ds-idrepo-2 12m 13589Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1178m 834Mi 08:24:25 DEBUG --- stderr --- 08:24:25 DEBUG 08:24:37 INFO 08:24:37 INFO [loop_until]: kubectl --namespace=xlou top node 08:24:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:24:37 INFO [loop_until]: OK (rc = 0) 08:24:37 DEBUG --- stdout --- 08:24:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4683m 29% 11579Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4948Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 6423m 40% 11696Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7664m 48% 12031Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 146m 0% 3122Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 83m 0% 1354Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 81m 0% 5303Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 61m 0% 14704Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 7328m 46% 15201Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2136m 13% 14839Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14637Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 6696m 42% 15143Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 7220m 45% 15204Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1270m 7% 2713Mi 4% 08:24:37 DEBUG --- stderr --- 08:24:37 DEBUG 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567489" 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567489" 08:24:49 INFO [http_cmd]: http status code OK 08:24:49 DEBUG --- status code --- 08:24:49 DEBUG http status code is 200 (expected 200) 08:24:49 DEBUG --- http response --- 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567489" 08:24:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:49 DEBUG ----- output ----- 08:24:49 DEBUG { 08:24:49 DEBUG "status": "success", 08:24:49 DEBUG "data": { 08:24:49 DEBUG "resultType": "vector", 08:24:49 DEBUG "result": [ 08:24:49 DEBUG { 08:24:49 DEBUG "metric": { 08:24:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG [...] 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 1684567489, 08:24:49 DEBUG "685.7333333333333" 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG } 08:24:49 INFO [http_cmd]: http status code OK 08:24:49 DEBUG --- status code --- 08:24:49 DEBUG http status code is 200 (expected 200) 08:24:49 DEBUG --- http response --- 08:24:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:49 DEBUG ----- output ----- 08:24:49 DEBUG { 08:24:49 DEBUG "status": "success", 08:24:49 DEBUG "data": { 08:24:49 DEBUG "resultType": "vector", 08:24:49 DEBUG "result": [ 08:24:49 DEBUG { 08:24:49 DEBUG "metric": { 08:24:49 DEBUG "pod": "ds-idrepo-2" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG [...] 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG "pod": "ds-idrepo-1" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 1684567489, 08:24:49 DEBUG "0" 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG } 08:24:49 INFO [http_cmd]: http status code OK 08:24:49 DEBUG --- status code --- 08:24:49 DEBUG http status code is 200 (expected 200) 08:24:49 DEBUG --- http response --- 08:24:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:24:49 DEBUG ----- output ----- 08:24:49 DEBUG { 08:24:49 DEBUG "status": "success", 08:24:49 DEBUG "data": { 08:24:49 DEBUG "resultType": "vector", 08:24:49 DEBUG "result": [ 08:24:49 DEBUG { 08:24:49 DEBUG "metric": { 08:24:49 DEBUG "pod": "ds-cts-2" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG [...] 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG "pod": "ds-idrepo-1" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 1684567489, 08:24:49 DEBUG "0" 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG } 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567489" 08:24:49 INFO [http_cmd]: http status code OK 08:24:49 DEBUG --- status code --- 08:24:49 DEBUG http status code is 200 (expected 200) 08:24:49 DEBUG --- http response --- 08:24:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:24:49 DEBUG ----- output ----- 08:24:49 DEBUG { 08:24:49 DEBUG "status": "success", 08:24:49 DEBUG "data": { 08:24:49 DEBUG "resultType": "vector", 08:24:49 DEBUG "result": [ 08:24:49 DEBUG { 08:24:49 DEBUG "metric": { 08:24:49 DEBUG "pod": "ds-cts-2" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG [...] 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG "pod": "ds-idrepo-1" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 1684567489, 08:24:49 DEBUG "0" 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG } 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567489" 08:24:49 INFO [http_cmd]: http status code OK 08:24:49 DEBUG --- status code --- 08:24:49 DEBUG http status code is 200 (expected 200) 08:24:49 DEBUG --- http response --- 08:24:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:49 DEBUG ----- output ----- 08:24:49 DEBUG { 08:24:49 DEBUG "status": "success", 08:24:49 DEBUG "data": { 08:24:49 DEBUG "resultType": "vector", 08:24:49 DEBUG "result": [ 08:24:49 DEBUG { 08:24:49 DEBUG "metric": { 08:24:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG [...] 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG "pod": "ds-idrepo-1" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 1684567489, 08:24:49 DEBUG "440201968.57292575" 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG } 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567489" 08:24:49 INFO [http_cmd]: http status code OK 08:24:49 DEBUG --- status code --- 08:24:49 DEBUG http status code is 200 (expected 200) 08:24:49 DEBUG --- http response --- 08:24:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:49 DEBUG ----- output ----- 08:24:49 DEBUG { 08:24:49 DEBUG "status": "success", 08:24:49 DEBUG "data": { 08:24:49 DEBUG "resultType": "vector", 08:24:49 DEBUG "result": [ 08:24:49 DEBUG { 08:24:49 DEBUG "metric": { 08:24:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG [...] 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 1684567489, 08:24:49 DEBUG "685.7666666666667" 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG } 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567489" 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567489" 08:24:49 INFO [http_cmd]: http status code OK 08:24:49 DEBUG --- status code --- 08:24:49 DEBUG http status code is 200 (expected 200) 08:24:49 DEBUG --- http response --- 08:24:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:49 DEBUG ----- output ----- 08:24:49 DEBUG { 08:24:49 DEBUG "status": "success", 08:24:49 DEBUG "data": { 08:24:49 DEBUG "resultType": "vector", 08:24:49 DEBUG "result": [ 08:24:49 DEBUG { 08:24:49 DEBUG "metric": { 08:24:49 DEBUG "pod": "ds-idrepo-2" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG [...] 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG "pod": "ds-idrepo-1" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 1684567489, 08:24:49 DEBUG "0" 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG } 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567489" 08:24:49 INFO [http_cmd]: http status code OK 08:24:49 DEBUG --- status code --- 08:24:49 DEBUG http status code is 200 (expected 200) 08:24:49 DEBUG --- http response --- 08:24:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:49 DEBUG ----- output ----- 08:24:49 DEBUG { 08:24:49 DEBUG "status": "success", 08:24:49 DEBUG "data": { 08:24:49 DEBUG "resultType": "vector", 08:24:49 DEBUG "result": [ 08:24:49 DEBUG { 08:24:49 DEBUG "metric": { 08:24:49 DEBUG "pod": "ds-idrepo-2" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG [...] 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG "pod": "ds-idrepo-1" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 1684567489, 08:24:49 DEBUG "0" 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG ] 08:24:49 DEBUG } 08:24:49 DEBUG } 08:24:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567489" 08:24:49 INFO [http_cmd]: http status code OK 08:24:49 DEBUG --- status code --- 08:24:49 DEBUG http status code is 200 (expected 200) 08:24:49 DEBUG --- http response --- 08:24:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:24:49 DEBUG ----- output ----- 08:24:49 DEBUG { 08:24:49 DEBUG "status": "success", 08:24:49 DEBUG "data": { 08:24:49 DEBUG "resultType": "vector", 08:24:49 DEBUG "result": [ 08:24:49 DEBUG { 08:24:49 DEBUG "metric": { 08:24:49 DEBUG "pod": "ds-cts-2" 08:24:49 DEBUG }, 08:24:49 DEBUG "value": [ 08:24:49 DEBUG 08:24:49 DEBUG 08:24:49 DEBUG 08:24:50 DEBUG [...] 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG "pod": "ds-idrepo-1" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 1684567489, 08:24:50 DEBUG "0" 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG { 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "data": { 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "result": [ 08:24:50 DEBUG { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "instance": "10.142.0.17:9100" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG [...] 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG "instance": "10.142.0.61:9100" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 1684567489, 08:24:50 DEBUG "7001022.2" 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567490" 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG { 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "data": { 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "result": [ 08:24:50 DEBUG { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "instance": "10.142.0.17:9100" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG [...] 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG "instance": "10.142.0.61:9100" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG "0.0015666666666675154" 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567490" 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567490" 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567490" 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG { 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "data": { 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "result": [ 08:24:50 DEBUG { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG [...] 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG "228.56666666666666" 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567490" 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567490" 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG { 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "data": { 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "result": [ 08:24:50 DEBUG { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "instance": "10.142.0.17:9100" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG [...] 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:50 DEBUG 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG "instance": "10.142.0.61:9100" 08:24:50 DEBUG { 08:24:50 DEBUG }, 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "value": [ 08:24:50 DEBUG "data": { 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "6999355.199999999" 08:24:50 DEBUG "result": [ 08:24:50 DEBUG ] 08:24:50 DEBUG { 08:24:50 DEBUG } 08:24:50 DEBUG "metric": { 08:24:50 DEBUG ] 08:24:50 DEBUG "instance": "10.142.0.17:9100" 08:24:50 DEBUG } 08:24:50 DEBUG }, 08:24:50 DEBUG } 08:24:50 DEBUG "value": [ 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG [...] 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:24:50 DEBUG 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG 08:24:50 DEBUG { 08:24:50 DEBUG { 08:24:50 DEBUG 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "instance": "10.142.0.61:9100" 08:24:50 DEBUG "data": { 08:24:50 DEBUG "data": { 08:24:50 DEBUG }, 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "value": [ 08:24:50 DEBUG "result": [ 08:24:50 DEBUG "result": [ 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG { 08:24:50 DEBUG { 08:24:50 DEBUG "0.0006666666666666524" 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG ] 08:24:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:24:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:24:50 DEBUG } 08:24:50 DEBUG }, 08:24:50 DEBUG }, 08:24:50 DEBUG ] 08:24:50 DEBUG "value": [ 08:24:50 DEBUG "value": [ 08:24:50 DEBUG } 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG } 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG [...] 08:24:50 DEBUG [...] 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:24:50 DEBUG "pod": "ds-idrepo-1" 08:24:50 DEBUG }, 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG "228.6" 08:24:50 DEBUG "0" 08:24:50 DEBUG ] 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 DEBUG ] 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567490" 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG { 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "data": { 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "result": [ 08:24:50 DEBUG { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG [...] 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG "14586413056" 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567490" 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG { 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "data": { 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "result": [ 08:24:50 DEBUG { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG [...] 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG "0.2" 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567490" 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567490" 08:24:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567490" 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG { 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "data": { 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "result": [ 08:24:50 DEBUG { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG [...] 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG "pod": "ds-idrepo-1" 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG "0.006415017655297999" 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 INFO [http_cmd]: http status code OK 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG --- status code --- 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG http status code is 200 (expected 200) 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG --- http response --- 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG ----- output ----- 08:24:50 DEBUG { 08:24:50 DEBUG { 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "status": "success", 08:24:50 DEBUG "data": { 08:24:50 DEBUG "data": { 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "resultType": "vector", 08:24:50 DEBUG "result": [ 08:24:50 DEBUG "result": [ 08:24:50 DEBUG { 08:24:50 DEBUG { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "metric": { 08:24:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:24:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:24:50 DEBUG }, 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG [...] 08:24:50 DEBUG [...] 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG 08:24:50 DEBUG "pod": "ds-idrepo-1" 08:24:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:24:50 DEBUG }, 08:24:50 DEBUG }, 08:24:50 DEBUG "value": [ 08:24:50 DEBUG "value": [ 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG 1684567490, 08:24:50 DEBUG "0.48221168905348877" 08:24:50 DEBUG "1.1840452634787921" 08:24:50 DEBUG ] 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 DEBUG ] 08:24:50 DEBUG ] 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 DEBUG } 08:24:50 DEBUG } 08:25:25 INFO 08:25:25 INFO [loop_until]: kubectl --namespace=xlou top pods 08:25:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:25:25 INFO [loop_until]: OK (rc = 0) 08:25:25 DEBUG --- stdout --- 08:25:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 5879m 9428Mi am-7849cf7bdb-lxfkl 3864m 9368Mi am-7849cf7bdb-td7js 4782m 9426Mi ds-cts-0 7104m 13882Mi ds-cts-1 6852m 13804Mi ds-cts-2 7546m 13872Mi ds-idrepo-0 2066m 13737Mi ds-idrepo-1 12m 13622Mi ds-idrepo-2 16m 13589Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1188m 835Mi 08:25:25 DEBUG --- stderr --- 08:25:25 DEBUG 08:25:37 INFO 08:25:37 INFO [loop_until]: kubectl --namespace=xlou top node 08:25:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:25:37 INFO [loop_until]: OK (rc = 0) 08:25:37 DEBUG --- stdout --- 08:25:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4457m 28% 11581Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 70m 0% 4949Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 5981m 37% 11694Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 8225m 51% 12028Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 152m 0% 3113Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1358Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 81m 0% 5301Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 61m 0% 14703Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 6502m 40% 15123Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 2141m 13% 14840Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 64m 0% 14637Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 7339m 46% 15194Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 6738m 42% 15198Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1283m 8% 2712Mi 4% 08:25:37 DEBUG --- stderr --- 08:25:37 DEBUG 08:25:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567549" 08:25:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567549" 08:25:49 INFO [http_cmd]: http status code OK 08:25:49 DEBUG --- status code --- 08:25:49 DEBUG http status code is 200 (expected 200) 08:25:49 DEBUG --- http response --- 08:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:49 DEBUG ----- output ----- 08:25:49 DEBUG { 08:25:49 DEBUG "status": "success", 08:25:49 DEBUG "data": { 08:25:49 DEBUG "resultType": "vector", 08:25:49 DEBUG "result": [ 08:25:49 DEBUG { 08:25:49 DEBUG "metric": { 08:25:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:25:49 DEBUG }, 08:25:49 DEBUG "value": [ 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG [...] 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:25:49 DEBUG }, 08:25:49 DEBUG "value": [ 08:25:49 DEBUG 1684567549, 08:25:49 DEBUG "111.23333333333333" 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG } 08:25:49 INFO [http_cmd]: http status code OK 08:25:49 DEBUG --- status code --- 08:25:49 DEBUG http status code is 200 (expected 200) 08:25:49 DEBUG --- http response --- 08:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:49 DEBUG ----- output ----- 08:25:49 DEBUG { 08:25:49 DEBUG "status": "success", 08:25:49 DEBUG "data": { 08:25:49 DEBUG "resultType": "vector", 08:25:49 DEBUG "result": [ 08:25:49 DEBUG { 08:25:49 DEBUG "metric": { 08:25:49 DEBUG "pod": "ds-idrepo-2" 08:25:49 DEBUG }, 08:25:49 DEBUG "value": [ 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG [...] 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG "pod": "ds-idrepo-1" 08:25:49 DEBUG }, 08:25:49 DEBUG "value": [ 08:25:49 DEBUG 1684567549, 08:25:49 DEBUG "0" 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG } 08:25:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567549" 08:25:49 INFO [http_cmd]: http status code OK 08:25:49 DEBUG --- status code --- 08:25:49 DEBUG http status code is 200 (expected 200) 08:25:49 DEBUG --- http response --- 08:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:25:49 DEBUG ----- output ----- 08:25:49 DEBUG { 08:25:49 DEBUG "status": "success", 08:25:49 DEBUG "data": { 08:25:49 DEBUG "resultType": "vector", 08:25:49 DEBUG "result": [ 08:25:49 DEBUG { 08:25:49 DEBUG "metric": { 08:25:49 DEBUG "pod": "ds-cts-2" 08:25:49 DEBUG }, 08:25:49 DEBUG "value": [ 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG [...] 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG "pod": "ds-idrepo-1" 08:25:49 DEBUG }, 08:25:49 DEBUG "value": [ 08:25:49 DEBUG 1684567549, 08:25:49 DEBUG "0" 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG } 08:25:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567549" 08:25:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567549" 08:25:49 INFO [http_cmd]: http status code OK 08:25:49 DEBUG --- status code --- 08:25:49 DEBUG http status code is 200 (expected 200) 08:25:49 DEBUG --- http response --- 08:25:49 INFO [http_cmd]: http status code OK 08:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:25:49 DEBUG --- status code --- 08:25:49 DEBUG ----- output ----- 08:25:49 DEBUG http status code is 200 (expected 200) 08:25:49 DEBUG { 08:25:49 DEBUG --- http response --- 08:25:49 DEBUG "status": "success", 08:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:25:49 DEBUG "data": { 08:25:49 DEBUG ----- output ----- 08:25:49 DEBUG "resultType": "vector", 08:25:49 DEBUG { 08:25:49 DEBUG "result": [ 08:25:49 DEBUG "status": "success", 08:25:49 DEBUG { 08:25:49 DEBUG "data": { 08:25:49 DEBUG "metric": { 08:25:49 DEBUG "resultType": "vector", 08:25:49 DEBUG "pod": "ds-cts-2" 08:25:49 DEBUG "result": [ 08:25:49 DEBUG }, 08:25:49 DEBUG { 08:25:49 DEBUG "value": [ 08:25:49 DEBUG "metric": { 08:25:49 DEBUG 08:25:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:25:49 DEBUG 08:25:49 DEBUG }, 08:25:49 DEBUG 08:25:49 DEBUG "value": [ 08:25:49 DEBUG [...] 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG [...] 08:25:49 DEBUG "pod": "ds-idrepo-1" 08:25:49 DEBUG 08:25:49 DEBUG }, 08:25:49 DEBUG 08:25:49 DEBUG "value": [ 08:25:49 DEBUG 08:25:49 DEBUG 1684567549, 08:25:49 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:25:49 DEBUG "0" 08:25:49 DEBUG }, 08:25:49 DEBUG ] 08:25:49 DEBUG "value": [ 08:25:49 DEBUG } 08:25:49 DEBUG 1684567549, 08:25:49 DEBUG ] 08:25:49 DEBUG "481186522.1845211" 08:25:49 DEBUG } 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG } 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG } 08:25:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567549" 08:25:49 INFO [http_cmd]: http status code OK 08:25:49 DEBUG --- status code --- 08:25:49 DEBUG http status code is 200 (expected 200) 08:25:49 DEBUG --- http response --- 08:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:49 DEBUG ----- output ----- 08:25:49 DEBUG { 08:25:49 DEBUG "status": "success", 08:25:49 DEBUG "data": { 08:25:49 DEBUG "resultType": "vector", 08:25:49 DEBUG "result": [ 08:25:49 DEBUG { 08:25:49 DEBUG "metric": { 08:25:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:25:49 DEBUG }, 08:25:49 DEBUG "value": [ 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG [...] 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG 08:25:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:25:49 DEBUG }, 08:25:49 DEBUG "value": [ 08:25:49 DEBUG 1684567549, 08:25:49 DEBUG "111.23333333333333" 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG ] 08:25:49 DEBUG } 08:25:49 DEBUG } 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567550" 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "pod": "ds-idrepo-2" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "pod": "ds-idrepo-1" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "0" 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567550" 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "pod": "ds-idrepo-2" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "pod": "ds-idrepo-1" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "0" 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567550" 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567550" 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567550" 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "pod": "ds-cts-2" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "pod": "ds-idrepo-1" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "0" 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG { 08:25:50 DEBUG { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "data": { 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "result": [ 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:25:50 DEBUG "instance": "10.142.0.17:9100" 08:25:50 DEBUG }, 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:25:50 DEBUG "instance": "10.142.0.61:9100" 08:25:50 DEBUG }, 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "0" 08:25:50 DEBUG "4049326.133333334" 08:25:50 DEBUG ] 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567550" 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567550" 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG { 08:25:50 DEBUG { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "data": { 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "result": [ 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:25:50 DEBUG "instance": "10.142.0.17:9100" 08:25:50 DEBUG }, 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:25:50 DEBUG "instance": "10.142.0.61:9100" 08:25:50 DEBUG }, 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "37.06666666666666" 08:25:50 DEBUG "0.001000000000000038" 08:25:50 DEBUG ] 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567550" 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567550" 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567550" 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "instance": "10.142.0.17:9100" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "instance": "10.142.0.61:9100" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "4045583.033333333" 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG { 08:25:50 DEBUG { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "data": { 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "result": [ 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:25:50 DEBUG "instance": "10.142.0.17:9100" 08:25:50 DEBUG }, 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:25:50 DEBUG "instance": "10.142.0.61:9100" 08:25:50 DEBUG }, 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "37.166666666666664" 08:25:50 DEBUG "0.00033333333333350387" 08:25:50 DEBUG ] 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567550" 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567550" 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG }, 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG "value": [ 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:50 DEBUG "0.2" 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG ] 08:25:50 DEBUG { 08:25:50 DEBUG } 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG ] 08:25:50 DEBUG "data": { 08:25:50 DEBUG } 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG } 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "14626242560" 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567550" 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- http response --- 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567550" 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567550" 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "result": [ 08:25:50 DEBUG { 08:25:50 DEBUG "metric": { 08:25:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:25:50 DEBUG }, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "1.971095819618716" 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG --- status code --- 08:25:50 INFO [http_cmd]: http status code OK 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG --- status code --- 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG http status code is 200 (expected 200) 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:50 DEBUG --- http response --- 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:25:50 DEBUG { 08:25:50 DEBUG ----- output ----- 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG { 08:25:50 DEBUG "data": { 08:25:50 DEBUG "status": "success", 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG "data": { 08:25:50 DEBUG "result": [ 08:25:50 DEBUG "resultType": "vector", 08:25:50 DEBUG { 08:25:50 DEBUG "result": [ 08:25:50 DEBUG "metric": { 08:25:50 DEBUG { 08:25:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:25:50 DEBUG "metric": { 08:25:50 DEBUG }, 08:25:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:25:50 DEBUG "value": [ 08:25:50 DEBUG }, 08:25:50 DEBUG 08:25:50 DEBUG "value": [ 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG [...] 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG 08:25:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:25:50 DEBUG 08:25:50 DEBUG }, 08:25:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:25:50 DEBUG "value": [ 08:25:50 DEBUG }, 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG "value": [ 08:25:50 DEBUG "0.22374224892923353" 08:25:50 DEBUG 1684567550, 08:25:50 DEBUG ] 08:25:50 DEBUG "1.223163988708627" 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG ] 08:25:50 DEBUG } 08:25:50 DEBUG } 08:25:50 DEBUG } 08:26:25 INFO 08:26:25 INFO [loop_until]: kubectl --namespace=xlou top pods 08:26:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:26:25 INFO [loop_until]: OK (rc = 0) 08:26:25 DEBUG --- stdout --- 08:26:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 4Mi am-7849cf7bdb-j4rs8 364m 9428Mi am-7849cf7bdb-lxfkl 309m 9368Mi am-7849cf7bdb-td7js 318m 9426Mi ds-cts-0 660m 13879Mi ds-cts-1 968m 13881Mi ds-cts-2 648m 13875Mi ds-idrepo-0 235m 13736Mi ds-idrepo-1 13m 13622Mi ds-idrepo-2 16m 13589Mi end-user-ui-787cb4f6b4-s5n85 1m 3Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 2m 301Mi 08:26:25 DEBUG --- stderr --- 08:26:25 DEBUG 08:26:37 INFO 08:26:37 INFO [loop_until]: kubectl --namespace=xlou top node 08:26:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:26:37 INFO [loop_until]: OK (rc = 0) 08:26:37 DEBUG --- stdout --- 08:26:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 386m 2% 11581Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4949Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 460m 2% 11693Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 418m 2% 12027Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 156m 0% 3133Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 83m 0% 1357Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 78m 0% 5300Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 60m 0% 14704Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 1022m 6% 15207Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 280m 1% 14836Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14637Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 716m 4% 15187Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 920m 5% 15217Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 80m 0% 2189Mi 3% 08:26:37 DEBUG --- stderr --- 08:26:37 DEBUG 08:26:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567609" 08:26:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567609" 08:26:49 INFO [http_cmd]: http status code OK 08:26:49 DEBUG --- status code --- 08:26:49 DEBUG http status code is 200 (expected 200) 08:26:49 DEBUG --- http response --- 08:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:49 DEBUG ----- output ----- 08:26:49 DEBUG { 08:26:49 DEBUG "status": "success", 08:26:49 DEBUG "data": { 08:26:49 DEBUG "resultType": "vector", 08:26:49 DEBUG "result": [ 08:26:49 DEBUG { 08:26:49 DEBUG "metric": { 08:26:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:26:49 DEBUG }, 08:26:49 DEBUG "value": [ 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG [...] 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:26:49 DEBUG }, 08:26:49 DEBUG "value": [ 08:26:49 DEBUG 1684567609, 08:26:49 DEBUG "0.03333333333333333" 08:26:49 DEBUG ] 08:26:49 DEBUG } 08:26:49 DEBUG ] 08:26:49 DEBUG } 08:26:49 DEBUG } 08:26:49 INFO [http_cmd]: http status code OK 08:26:49 DEBUG --- status code --- 08:26:49 DEBUG http status code is 200 (expected 200) 08:26:49 DEBUG --- http response --- 08:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:49 DEBUG ----- output ----- 08:26:49 DEBUG { 08:26:49 DEBUG "status": "success", 08:26:49 DEBUG "data": { 08:26:49 DEBUG "resultType": "vector", 08:26:49 DEBUG "result": [ 08:26:49 DEBUG { 08:26:49 DEBUG "metric": { 08:26:49 DEBUG "pod": "ds-idrepo-2" 08:26:49 DEBUG }, 08:26:49 DEBUG "value": [ 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG [...] 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG "pod": "ds-idrepo-1" 08:26:49 DEBUG }, 08:26:49 DEBUG "value": [ 08:26:49 DEBUG 1684567609, 08:26:49 DEBUG "0" 08:26:49 DEBUG ] 08:26:49 DEBUG } 08:26:49 DEBUG ] 08:26:49 DEBUG } 08:26:49 DEBUG } 08:26:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567609" 08:26:49 INFO [http_cmd]: http status code OK 08:26:49 DEBUG --- status code --- 08:26:49 DEBUG http status code is 200 (expected 200) 08:26:49 DEBUG --- http response --- 08:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:26:49 DEBUG ----- output ----- 08:26:49 DEBUG { 08:26:49 DEBUG "status": "success", 08:26:49 DEBUG "data": { 08:26:49 DEBUG "resultType": "vector", 08:26:49 DEBUG "result": [ 08:26:49 DEBUG { 08:26:49 DEBUG "metric": { 08:26:49 DEBUG "pod": "ds-cts-2" 08:26:49 DEBUG }, 08:26:49 DEBUG "value": [ 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG [...] 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG "pod": "ds-idrepo-1" 08:26:49 DEBUG }, 08:26:49 DEBUG "value": [ 08:26:49 DEBUG 1684567609, 08:26:49 DEBUG "0" 08:26:49 DEBUG ] 08:26:49 DEBUG } 08:26:49 DEBUG ] 08:26:49 DEBUG } 08:26:49 DEBUG } 08:26:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567609" 08:26:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567609" 08:26:49 INFO [http_cmd]: http status code OK 08:26:49 DEBUG --- status code --- 08:26:49 DEBUG http status code is 200 (expected 200) 08:26:49 DEBUG --- http response --- 08:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:26:49 DEBUG ----- output ----- 08:26:49 DEBUG { 08:26:49 INFO [http_cmd]: http status code OK 08:26:49 DEBUG "status": "success", 08:26:49 DEBUG --- status code --- 08:26:49 DEBUG "data": { 08:26:49 DEBUG http status code is 200 (expected 200) 08:26:49 DEBUG "resultType": "vector", 08:26:49 DEBUG --- http response --- 08:26:49 DEBUG "result": [ 08:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:26:49 DEBUG { 08:26:49 DEBUG ----- output ----- 08:26:49 DEBUG "metric": { 08:26:49 DEBUG { 08:26:49 DEBUG "pod": "ds-cts-2" 08:26:49 DEBUG "status": "success", 08:26:49 DEBUG }, 08:26:49 DEBUG "data": { 08:26:49 DEBUG "value": [ 08:26:49 DEBUG "resultType": "vector", 08:26:49 DEBUG 08:26:49 DEBUG "result": [ 08:26:49 DEBUG 08:26:49 DEBUG { 08:26:49 DEBUG 08:26:49 DEBUG "metric": { 08:26:49 DEBUG [...] 08:26:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:26:49 DEBUG 08:26:49 DEBUG }, 08:26:49 DEBUG 08:26:49 DEBUG "value": [ 08:26:49 DEBUG 08:26:49 DEBUG 08:26:49 DEBUG "pod": "ds-idrepo-1" 08:26:49 DEBUG 08:26:49 DEBUG }, 08:26:49 DEBUG 08:26:49 DEBUG "value": [ 08:26:49 DEBUG [...] 08:26:49 DEBUG 1684567609, 08:26:49 DEBUG 08:26:49 DEBUG "0" 08:26:49 DEBUG 08:26:49 DEBUG ] 08:26:49 DEBUG 08:26:49 DEBUG } 08:26:49 DEBUG "pod": "ds-idrepo-1" 08:26:49 DEBUG ] 08:26:49 DEBUG }, 08:26:49 DEBUG } 08:26:49 DEBUG "value": [ 08:26:49 DEBUG } 08:26:49 DEBUG 1684567609, 08:26:49 DEBUG "0" 08:26:49 DEBUG ] 08:26:49 DEBUG } 08:26:49 DEBUG ] 08:26:49 DEBUG } 08:26:49 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "0.03333333333333333" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "pod": "ds-cts-2" 08:26:50 DEBUG "pod": "ds-idrepo-2" 08:26:50 DEBUG }, 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "pod": "ds-idrepo-1" 08:26:50 DEBUG "pod": "ds-idrepo-1" 08:26:50 DEBUG }, 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "0" 08:26:50 DEBUG "0" 08:26:50 DEBUG ] 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "instance": "10.142.0.17:9100" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "instance": "10.142.0.61:9100" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "9059.966666666665" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "pod": "ds-idrepo-2" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "pod": "ds-idrepo-1" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "0" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "0" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:26:50 DEBUG "instance": "10.142.0.17:9100" 08:26:50 DEBUG }, 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:26:50 DEBUG "instance": "10.142.0.61:9100" 08:26:50 DEBUG }, 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "0.03333333333333333" 08:26:50 DEBUG "0.0005666666666665302" 08:26:50 DEBUG ] 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567610" 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "instance": "10.142.0.17:9100" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "instance": "10.142.0.61:9100" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "3264.766666666667" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "0" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "instance": "10.142.0.17:9100" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "instance": "10.142.0.61:9100" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "0" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "0.2" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "14580666368" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567610" 08:26:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567610" 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "data": { 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG [...] 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "pod": "ds-idrepo-1" 08:26:50 DEBUG }, 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG "0" 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:26:50 INFO [http_cmd]: http status code OK 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG --- status code --- 08:26:50 DEBUG { 08:26:50 DEBUG http status code is 200 (expected 200) 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG --- http response --- 08:26:50 DEBUG "data": { 08:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG ----- output ----- 08:26:50 DEBUG "result": [ 08:26:50 DEBUG { 08:26:50 DEBUG { 08:26:50 DEBUG "status": "success", 08:26:50 DEBUG "metric": { 08:26:50 DEBUG "data": { 08:26:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:26:50 DEBUG "resultType": "vector", 08:26:50 DEBUG }, 08:26:50 DEBUG "result": [ 08:26:50 DEBUG "value": [ 08:26:50 DEBUG { 08:26:50 DEBUG 08:26:50 DEBUG "metric": { 08:26:50 DEBUG 08:26:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:26:50 DEBUG 08:26:50 DEBUG }, 08:26:50 DEBUG [...] 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG 08:26:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:26:50 DEBUG [...] 08:26:50 DEBUG }, 08:26:50 DEBUG 08:26:50 DEBUG "value": [ 08:26:50 DEBUG 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG 08:26:50 DEBUG "0" 08:26:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:26:50 DEBUG ] 08:26:50 DEBUG }, 08:26:50 DEBUG } 08:26:50 DEBUG "value": [ 08:26:50 DEBUG ] 08:26:50 DEBUG 1684567610, 08:26:50 DEBUG } 08:26:50 DEBUG "0.028462589787935255" 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG ] 08:26:50 DEBUG } 08:26:50 DEBUG } 08:27:25 INFO 08:27:25 INFO [loop_until]: kubectl --namespace=xlou top pods 08:27:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:27:26 INFO [loop_until]: OK (rc = 0) 08:27:26 DEBUG --- stdout --- 08:27:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 347m 9397Mi am-7849cf7bdb-lxfkl 291m 9341Mi am-7849cf7bdb-td7js 311m 9403Mi ds-cts-0 654m 13828Mi ds-cts-1 732m 13770Mi ds-cts-2 971m 13844Mi ds-idrepo-0 695m 13663Mi ds-idrepo-1 353m 13554Mi ds-idrepo-2 261m 13522Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 5m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1119m 559Mi 08:27:26 DEBUG --- stderr --- 08:27:26 DEBUG 08:27:37 INFO 08:27:37 INFO [loop_until]: kubectl --namespace=xlou top node 08:27:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:27:37 INFO [loop_until]: OK (rc = 0) 08:27:37 DEBUG --- stdout --- 08:27:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 407m 2% 11560Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 75m 0% 4949Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 473m 2% 11671Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 402m 2% 12013Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 145m 0% 3133Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 89m 0% 1361Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 82m 0% 5295Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 367m 2% 14636Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1317m 8% 15105Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 751m 4% 14779Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 430m 2% 14574Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 1033m 6% 15163Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 998m 6% 15169Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1296m 8% 2523Mi 4% 08:27:37 DEBUG --- stderr --- 08:27:37 DEBUG 08:27:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567669" 08:27:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567669" 08:27:49 INFO [http_cmd]: http status code OK 08:27:49 INFO [http_cmd]: http status code OK 08:27:49 DEBUG --- status code --- 08:27:49 DEBUG --- status code --- 08:27:49 DEBUG http status code is 200 (expected 200) 08:27:49 DEBUG http status code is 200 (expected 200) 08:27:49 DEBUG --- http response --- 08:27:49 DEBUG --- http response --- 08:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:49 DEBUG ----- output ----- 08:27:49 DEBUG ----- output ----- 08:27:49 DEBUG { 08:27:49 DEBUG { 08:27:49 DEBUG "status": "success", 08:27:49 DEBUG "status": "success", 08:27:49 DEBUG "data": { 08:27:49 DEBUG "data": { 08:27:49 DEBUG "resultType": "vector", 08:27:49 DEBUG "resultType": "vector", 08:27:49 DEBUG "result": [ 08:27:49 DEBUG "result": [ 08:27:49 DEBUG { 08:27:49 DEBUG { 08:27:49 DEBUG "metric": { 08:27:49 DEBUG "metric": { 08:27:49 DEBUG "pod": "ds-idrepo-2" 08:27:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:27:49 DEBUG }, 08:27:49 DEBUG }, 08:27:49 DEBUG "value": [ 08:27:49 DEBUG "value": [ 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG [...] 08:27:49 DEBUG [...] 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG "pod": "ds-idrepo-1" 08:27:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:27:49 DEBUG }, 08:27:49 DEBUG }, 08:27:49 DEBUG "value": [ 08:27:49 DEBUG "value": [ 08:27:49 DEBUG 1684567669, 08:27:49 DEBUG 1684567669, 08:27:49 DEBUG "0" 08:27:49 DEBUG "0" 08:27:49 DEBUG ] 08:27:49 DEBUG ] 08:27:49 DEBUG } 08:27:49 DEBUG } 08:27:49 DEBUG ] 08:27:49 DEBUG ] 08:27:49 DEBUG } 08:27:49 DEBUG } 08:27:49 DEBUG } 08:27:49 DEBUG } 08:27:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567669" 08:27:49 INFO [http_cmd]: http status code OK 08:27:49 DEBUG --- status code --- 08:27:49 DEBUG http status code is 200 (expected 200) 08:27:49 DEBUG --- http response --- 08:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:27:49 DEBUG ----- output ----- 08:27:49 DEBUG { 08:27:49 DEBUG "status": "success", 08:27:49 DEBUG "data": { 08:27:49 DEBUG "resultType": "vector", 08:27:49 DEBUG "result": [ 08:27:49 DEBUG { 08:27:49 DEBUG "metric": { 08:27:49 DEBUG "pod": "ds-cts-2" 08:27:49 DEBUG }, 08:27:49 DEBUG "value": [ 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG [...] 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG "pod": "ds-idrepo-1" 08:27:49 DEBUG }, 08:27:49 DEBUG "value": [ 08:27:49 DEBUG 1684567669, 08:27:49 DEBUG "0" 08:27:49 DEBUG ] 08:27:49 DEBUG } 08:27:49 DEBUG ] 08:27:49 DEBUG } 08:27:49 DEBUG } 08:27:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567669" 08:27:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567669" 08:27:49 INFO [http_cmd]: http status code OK 08:27:49 DEBUG --- status code --- 08:27:49 INFO [http_cmd]: http status code OK 08:27:49 DEBUG http status code is 200 (expected 200) 08:27:49 DEBUG --- status code --- 08:27:49 DEBUG --- http response --- 08:27:49 DEBUG http status code is 200 (expected 200) 08:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:27:49 DEBUG --- http response --- 08:27:49 DEBUG ----- output ----- 08:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:27:49 DEBUG { 08:27:49 DEBUG ----- output ----- 08:27:49 DEBUG "status": "success", 08:27:49 DEBUG { 08:27:49 DEBUG "data": { 08:27:49 DEBUG "status": "success", 08:27:49 DEBUG "resultType": "vector", 08:27:49 DEBUG "data": { 08:27:49 DEBUG "result": [ 08:27:49 DEBUG "resultType": "vector", 08:27:49 DEBUG { 08:27:49 DEBUG "result": [ 08:27:49 DEBUG "metric": { 08:27:49 DEBUG { 08:27:49 DEBUG "pod": "ds-cts-2" 08:27:49 DEBUG "metric": { 08:27:49 DEBUG }, 08:27:49 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:27:49 DEBUG "value": [ 08:27:49 DEBUG }, 08:27:49 DEBUG 08:27:49 DEBUG "value": [ 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG [...] 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG [...] 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG 08:27:49 DEBUG "pod": "ds-idrepo-1" 08:27:49 DEBUG 08:27:49 DEBUG }, 08:27:49 DEBUG "pod": "ds-idrepo-1" 08:27:49 DEBUG "value": [ 08:27:49 DEBUG }, 08:27:49 DEBUG 1684567669, 08:27:49 DEBUG "value": [ 08:27:49 DEBUG "0" 08:27:49 DEBUG 1684567669, 08:27:49 DEBUG ] 08:27:49 DEBUG "0" 08:27:49 DEBUG } 08:27:49 DEBUG ] 08:27:49 DEBUG ] 08:27:49 DEBUG } 08:27:49 DEBUG } 08:27:49 DEBUG ] 08:27:49 DEBUG } 08:27:49 DEBUG } 08:27:49 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "ds-cts-2" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "ds-idrepo-1" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0.03333333333333333" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "ds-idrepo-2" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "ds-idrepo-1" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567670" 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "instance": "10.142.0.17:9100" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "instance": "10.142.0.61:9100" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "6864.233333333334" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "ds-idrepo-2" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "ds-idrepo-1" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "ds-idrepo-0" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0.1683048337148243" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "instance": "10.142.0.17:9100" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "instance": "10.142.0.61:9100" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0.6191000000000002" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567670" 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "instance": "10.142.0.17:9100" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "instance": "10.142.0.61:9100" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "5454.7" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "instance": "10.142.0.17:9100" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "instance": "10.142.0.61:9100" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0.001000000000000038" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0.2" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "14610432000" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "ds-idrepo-0" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "1.0771509357748754" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567670" 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "pod": "ds-idrepo-1" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:27:50 INFO [http_cmd]: http status code OK 08:27:50 DEBUG --- status code --- 08:27:50 DEBUG http status code is 200 (expected 200) 08:27:50 DEBUG --- http response --- 08:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:27:50 DEBUG ----- output ----- 08:27:50 DEBUG { 08:27:50 DEBUG "status": "success", 08:27:50 DEBUG "data": { 08:27:50 DEBUG "resultType": "vector", 08:27:50 DEBUG "result": [ 08:27:50 DEBUG { 08:27:50 DEBUG "metric": { 08:27:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG [...] 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG 08:27:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:27:50 DEBUG }, 08:27:50 DEBUG "value": [ 08:27:50 DEBUG 1684567670, 08:27:50 DEBUG "0.8650147902111804" 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG ] 08:27:50 DEBUG } 08:27:50 DEBUG } 08:28:26 INFO 08:28:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:28:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:28:26 INFO [loop_until]: OK (rc = 0) 08:28:26 DEBUG --- stdout --- 08:28:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 460m 9406Mi am-7849cf7bdb-lxfkl 305m 9343Mi am-7849cf7bdb-td7js 330m 9404Mi ds-cts-0 691m 13736Mi ds-cts-1 696m 13828Mi ds-cts-2 1127m 13874Mi ds-idrepo-0 233m 13661Mi ds-idrepo-1 11m 13554Mi ds-idrepo-2 19m 13520Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 991m 669Mi 08:28:26 DEBUG --- stderr --- 08:28:26 DEBUG 08:28:37 INFO 08:28:37 INFO [loop_until]: kubectl --namespace=xlou top node 08:28:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:28:37 INFO [loop_until]: OK (rc = 0) 08:28:37 DEBUG --- stdout --- 08:28:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 393m 2% 11559Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 75m 0% 4946Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 664m 4% 11677Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 425m 2% 12005Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 172m 1% 3083Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 83m 0% 1364Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 83m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14640Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 745m 4% 15173Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 289m 1% 14769Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14569Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 975m 6% 15197Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 755m 4% 15076Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1092m 6% 2695Mi 4% 08:28:37 DEBUG --- stderr --- 08:28:37 DEBUG 08:28:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567729" 08:28:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567729" 08:28:49 INFO [http_cmd]: http status code OK 08:28:49 INFO [http_cmd]: http status code OK 08:28:49 DEBUG --- status code --- 08:28:49 DEBUG --- status code --- 08:28:49 DEBUG http status code is 200 (expected 200) 08:28:49 DEBUG http status code is 200 (expected 200) 08:28:49 DEBUG --- http response --- 08:28:49 DEBUG --- http response --- 08:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:49 DEBUG ----- output ----- 08:28:49 DEBUG ----- output ----- 08:28:49 DEBUG { 08:28:49 DEBUG { 08:28:49 DEBUG "status": "success", 08:28:49 DEBUG "status": "success", 08:28:49 DEBUG "data": { 08:28:49 DEBUG "data": { 08:28:49 DEBUG "resultType": "vector", 08:28:49 DEBUG "resultType": "vector", 08:28:49 DEBUG "result": [ 08:28:49 DEBUG "result": [ 08:28:49 DEBUG { 08:28:49 DEBUG { 08:28:49 DEBUG "metric": { 08:28:49 DEBUG "metric": { 08:28:49 DEBUG "pod": "ds-idrepo-2" 08:28:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:28:49 DEBUG }, 08:28:49 DEBUG }, 08:28:49 DEBUG "value": [ 08:28:49 DEBUG "value": [ 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG [...] 08:28:49 DEBUG [...] 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG "pod": "ds-idrepo-1" 08:28:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:28:49 DEBUG }, 08:28:49 DEBUG }, 08:28:49 DEBUG "value": [ 08:28:49 DEBUG "value": [ 08:28:49 DEBUG 1684567729, 08:28:49 DEBUG 1684567729, 08:28:49 DEBUG "0" 08:28:49 DEBUG "0" 08:28:49 DEBUG ] 08:28:49 DEBUG ] 08:28:49 DEBUG } 08:28:49 DEBUG } 08:28:49 DEBUG ] 08:28:49 DEBUG ] 08:28:49 DEBUG } 08:28:49 DEBUG } 08:28:49 DEBUG } 08:28:49 DEBUG } 08:28:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567729" 08:28:49 INFO [http_cmd]: http status code OK 08:28:49 DEBUG --- status code --- 08:28:49 DEBUG http status code is 200 (expected 200) 08:28:49 DEBUG --- http response --- 08:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:28:49 DEBUG ----- output ----- 08:28:49 DEBUG { 08:28:49 DEBUG "status": "success", 08:28:49 DEBUG "data": { 08:28:49 DEBUG "resultType": "vector", 08:28:49 DEBUG "result": [ 08:28:49 DEBUG { 08:28:49 DEBUG "metric": { 08:28:49 DEBUG "pod": "ds-cts-2" 08:28:49 DEBUG }, 08:28:49 DEBUG "value": [ 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG [...] 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG 08:28:49 DEBUG "pod": "ds-idrepo-1" 08:28:49 DEBUG }, 08:28:49 DEBUG "value": [ 08:28:49 DEBUG 1684567729, 08:28:49 DEBUG "0" 08:28:49 DEBUG ] 08:28:49 DEBUG } 08:28:49 DEBUG ] 08:28:49 DEBUG } 08:28:49 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "ds-idrepo-2" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "415759037.04527754" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "ds-cts-2" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "ds-idrepo-1" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "ds-cts-2" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "ds-idrepo-1" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "ds-idrepo-2" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "ds-idrepo-1" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "ds-idrepo-2" 08:28:50 DEBUG }, 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567730" 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "ds-idrepo-1" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG { 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:50 DEBUG "data": { 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG { 08:28:50 DEBUG "result": [ 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG { 08:28:50 DEBUG "data": { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "instance": "10.142.0.17:9100" 08:28:50 DEBUG "result": [ 08:28:50 DEBUG }, 08:28:50 DEBUG { 08:28:50 DEBUG "value": [ 08:28:50 DEBUG "metric": { 08:28:50 DEBUG 08:28:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:28:50 DEBUG 08:28:50 DEBUG }, 08:28:50 DEBUG 08:28:50 DEBUG "value": [ 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG "instance": "10.142.0.61:9100" 08:28:50 DEBUG 08:28:50 DEBUG }, 08:28:50 DEBUG 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:28:50 DEBUG "6539.066666666667" 08:28:50 DEBUG }, 08:28:50 DEBUG ] 08:28:50 DEBUG "value": [ 08:28:50 DEBUG } 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG ] 08:28:50 DEBUG "0" 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "ds-idrepo-2" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "instance": "10.142.0.17:9100" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "instance": "10.142.0.61:9100" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0.00046666666666700016" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567730" 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG { 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:50 DEBUG "data": { 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG { 08:28:50 DEBUG "result": [ 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG { 08:28:50 DEBUG "data": { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "instance": "10.142.0.17:9100" 08:28:50 DEBUG "result": [ 08:28:50 DEBUG }, 08:28:50 DEBUG { 08:28:50 DEBUG "value": [ 08:28:50 DEBUG "metric": { 08:28:50 DEBUG 08:28:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:28:50 DEBUG 08:28:50 DEBUG }, 08:28:50 DEBUG 08:28:50 DEBUG "value": [ 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG "instance": "10.142.0.61:9100" 08:28:50 DEBUG 08:28:50 DEBUG }, 08:28:50 DEBUG 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:28:50 DEBUG "2894.6666666666665" 08:28:50 DEBUG }, 08:28:50 DEBUG ] 08:28:50 DEBUG "value": [ 08:28:50 DEBUG } 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG ] 08:28:50 DEBUG "0" 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "instance": "10.142.0.17:9100" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "instance": "10.142.0.61:9100" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0.2" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "14573207552" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567730" 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "1.1062171248946289" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "ds-idrepo-2" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0.016831436422271816" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:28:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567730" 08:28:50 INFO [http_cmd]: http status code OK 08:28:50 DEBUG --- status code --- 08:28:50 DEBUG http status code is 200 (expected 200) 08:28:50 DEBUG --- http response --- 08:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:28:50 DEBUG ----- output ----- 08:28:50 DEBUG { 08:28:50 DEBUG "status": "success", 08:28:50 DEBUG "data": { 08:28:50 DEBUG "resultType": "vector", 08:28:50 DEBUG "result": [ 08:28:50 DEBUG { 08:28:50 DEBUG "metric": { 08:28:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG [...] 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG 08:28:50 DEBUG "pod": "ds-idrepo-2" 08:28:50 DEBUG }, 08:28:50 DEBUG "value": [ 08:28:50 DEBUG 1684567730, 08:28:50 DEBUG "0.9177884080542885" 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG ] 08:28:50 DEBUG } 08:28:50 DEBUG } 08:29:26 INFO 08:29:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:29:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:29:26 INFO [loop_until]: OK (rc = 0) 08:29:26 DEBUG --- stdout --- 08:29:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 413m 9407Mi am-7849cf7bdb-lxfkl 312m 9343Mi am-7849cf7bdb-td7js 307m 9404Mi ds-cts-0 832m 13783Mi ds-cts-1 1602m 13742Mi ds-cts-2 677m 13763Mi ds-idrepo-0 238m 13661Mi ds-idrepo-1 14m 13554Mi ds-idrepo-2 14m 13520Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 5m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 991m 901Mi 08:29:26 DEBUG --- stderr --- 08:29:26 DEBUG 08:29:38 INFO 08:29:38 INFO [loop_until]: kubectl --namespace=xlou top node 08:29:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:29:38 INFO [loop_until]: OK (rc = 0) 08:29:38 DEBUG --- stdout --- 08:29:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 392m 2% 11558Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 70m 0% 4945Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 572m 3% 11676Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 410m 2% 12005Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 159m 1% 3125Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1362Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 83m 0% 5296Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 62m 0% 14640Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1627m 10% 15078Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 287m 1% 14767Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14571Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 713m 4% 15094Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 923m 5% 15117Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1082m 6% 2911Mi 4% 08:29:38 DEBUG --- stderr --- 08:29:38 DEBUG 08:29:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567789" 08:29:49 INFO [http_cmd]: http status code OK 08:29:49 DEBUG --- status code --- 08:29:49 DEBUG http status code is 200 (expected 200) 08:29:49 DEBUG --- http response --- 08:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:49 DEBUG ----- output ----- 08:29:49 DEBUG { 08:29:49 DEBUG "status": "success", 08:29:49 DEBUG "data": { 08:29:49 DEBUG "resultType": "vector", 08:29:49 DEBUG "result": [ 08:29:49 DEBUG { 08:29:49 DEBUG "metric": { 08:29:49 DEBUG "pod": "ds-idrepo-2" 08:29:49 DEBUG }, 08:29:49 DEBUG "value": [ 08:29:49 DEBUG 08:29:49 DEBUG 08:29:49 DEBUG 08:29:49 DEBUG [...] 08:29:49 DEBUG 08:29:49 DEBUG 08:29:49 DEBUG 08:29:49 DEBUG "pod": "ds-idrepo-1" 08:29:49 DEBUG }, 08:29:49 DEBUG "value": [ 08:29:49 DEBUG 1684567789, 08:29:49 DEBUG "0" 08:29:49 DEBUG ] 08:29:49 DEBUG } 08:29:49 DEBUG ] 08:29:49 DEBUG } 08:29:49 DEBUG } 08:29:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567789" 08:29:49 INFO [http_cmd]: http status code OK 08:29:49 DEBUG --- status code --- 08:29:49 DEBUG http status code is 200 (expected 200) 08:29:49 DEBUG --- http response --- 08:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:49 DEBUG ----- output ----- 08:29:49 DEBUG { 08:29:49 DEBUG "status": "success", 08:29:49 DEBUG "data": { 08:29:49 DEBUG "resultType": "vector", 08:29:49 DEBUG "result": [ 08:29:49 DEBUG { 08:29:49 DEBUG "metric": { 08:29:49 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:29:49 DEBUG }, 08:29:49 DEBUG "value": [ 08:29:49 DEBUG 08:29:49 DEBUG 08:29:49 DEBUG 08:29:49 DEBUG [...] 08:29:49 DEBUG 08:29:49 DEBUG 08:29:49 DEBUG 08:29:49 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:29:49 DEBUG }, 08:29:49 DEBUG "value": [ 08:29:49 DEBUG 1684567789, 08:29:49 DEBUG "0" 08:29:49 DEBUG ] 08:29:49 DEBUG } 08:29:49 DEBUG ] 08:29:49 DEBUG } 08:29:49 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "pod": "ds-cts-2" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "ds-idrepo-1" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:29:50 DEBUG { 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG { 08:29:50 DEBUG "data": { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "result": [ 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG { 08:29:50 DEBUG "result": [ 08:29:50 DEBUG "metric": { 08:29:50 DEBUG { 08:29:50 DEBUG "pod": "ds-cts-2" 08:29:50 DEBUG "metric": { 08:29:50 DEBUG }, 08:29:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:29:50 DEBUG "value": [ 08:29:50 DEBUG }, 08:29:50 DEBUG 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "ds-idrepo-1" 08:29:50 DEBUG 08:29:50 DEBUG }, 08:29:50 DEBUG "pod": "ds-idrepo-1" 08:29:50 DEBUG "value": [ 08:29:50 DEBUG }, 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG "0" 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG ] 08:29:50 DEBUG "0" 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "pod": "ds-idrepo-2" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "ds-idrepo-1" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "pod": "ds-cts-2" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "ds-idrepo-1" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG { 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:50 DEBUG "data": { 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG { 08:29:50 DEBUG "result": [ 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG { 08:29:50 DEBUG "data": { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "pod": "ds-idrepo-2" 08:29:50 DEBUG "result": [ 08:29:50 DEBUG }, 08:29:50 DEBUG { 08:29:50 DEBUG "value": [ 08:29:50 DEBUG "metric": { 08:29:50 DEBUG 08:29:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:29:50 DEBUG 08:29:50 DEBUG }, 08:29:50 DEBUG 08:29:50 DEBUG "value": [ 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG "pod": "ds-idrepo-1" 08:29:50 DEBUG 08:29:50 DEBUG }, 08:29:50 DEBUG 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "pod": "ds-cts-0" 08:29:50 DEBUG "0" 08:29:50 DEBUG }, 08:29:50 DEBUG ] 08:29:50 DEBUG "value": [ 08:29:50 DEBUG } 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG ] 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567790" 08:29:50 DEBUG "932.484632696248" 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "instance": "10.142.0.17:9100" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "instance": "10.142.0.61:9100" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "6723.966666666666" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "instance": "10.142.0.17:9100" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "instance": "10.142.0.61:9100" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0.00046666666666700016" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567790" 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567790" 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "instance": "10.142.0.17:9100" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "instance": "10.142.0.61:9100" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "3107.4" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "instance": "10.142.0.17:9100" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "instance": "10.142.0.61:9100" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0.2" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "14537486336" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "1.0275572552337584" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "ds-idrepo-1" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "0" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:29:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567790" 08:29:50 INFO [http_cmd]: http status code OK 08:29:50 DEBUG --- status code --- 08:29:50 DEBUG http status code is 200 (expected 200) 08:29:50 DEBUG --- http response --- 08:29:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:29:50 DEBUG ----- output ----- 08:29:50 DEBUG { 08:29:50 DEBUG "status": "success", 08:29:50 DEBUG "data": { 08:29:50 DEBUG "resultType": "vector", 08:29:50 DEBUG "result": [ 08:29:50 DEBUG { 08:29:50 DEBUG "metric": { 08:29:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG [...] 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG 08:29:50 DEBUG "pod": "ds-cts-0" 08:29:50 DEBUG }, 08:29:50 DEBUG "value": [ 08:29:50 DEBUG 1684567790, 08:29:50 DEBUG "19.8515333714017" 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG ] 08:29:50 DEBUG } 08:29:50 DEBUG } 08:30:26 INFO 08:30:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:30:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:30:26 INFO [loop_until]: OK (rc = 0) 08:30:26 DEBUG --- stdout --- 08:30:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 357m 9407Mi am-7849cf7bdb-lxfkl 314m 9343Mi am-7849cf7bdb-td7js 298m 9403Mi ds-cts-0 646m 13832Mi ds-cts-1 1125m 13769Mi ds-cts-2 612m 13839Mi ds-idrepo-0 229m 13661Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1006m 1098Mi 08:30:26 DEBUG --- stderr --- 08:30:26 DEBUG 08:30:38 INFO 08:30:38 INFO [loop_until]: kubectl --namespace=xlou top node 08:30:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:30:38 INFO [loop_until]: OK (rc = 0) 08:30:38 DEBUG --- stdout --- 08:30:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 404m 2% 11562Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 74m 0% 4946Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 470m 2% 11678Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 389m 2% 12008Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 154m 0% 3129Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 85m 0% 1362Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 83m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 59m 0% 14639Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1016m 6% 15117Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 278m 1% 14765Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14571Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 659m 4% 15170Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 722m 4% 15180Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1046m 6% 3136Mi 5% 08:30:38 DEBUG --- stderr --- 08:30:38 DEBUG 08:30:49 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567849" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "ds-idrepo-2" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "ds-idrepo-1" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567849, 08:30:50 DEBUG "0" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG { 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:30:50 DEBUG "data": { 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG { 08:30:50 DEBUG "result": [ 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG { 08:30:50 DEBUG "data": { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:30:50 DEBUG "result": [ 08:30:50 DEBUG }, 08:30:50 DEBUG { 08:30:50 DEBUG "value": [ 08:30:50 DEBUG "metric": { 08:30:50 DEBUG 08:30:50 DEBUG "pod": "ds-cts-2" 08:30:50 DEBUG 08:30:50 DEBUG }, 08:30:50 DEBUG 08:30:50 DEBUG "value": [ 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:30:50 DEBUG 08:30:50 DEBUG }, 08:30:50 DEBUG 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "pod": "ds-idrepo-1" 08:30:50 DEBUG "0" 08:30:50 DEBUG }, 08:30:50 DEBUG ] 08:30:50 DEBUG "value": [ 08:30:50 DEBUG } 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG ] 08:30:50 DEBUG "0" 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "ds-idrepo-0" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "933.971790612072" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "ds-cts-2" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "ds-idrepo-1" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG { 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:30:50 DEBUG "data": { 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG { 08:30:50 DEBUG "result": [ 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG { 08:30:50 DEBUG "data": { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "pod": "ds-idrepo-2" 08:30:50 DEBUG "result": [ 08:30:50 DEBUG }, 08:30:50 DEBUG { 08:30:50 DEBUG "value": [ 08:30:50 DEBUG "metric": { 08:30:50 DEBUG 08:30:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:30:50 DEBUG 08:30:50 DEBUG }, 08:30:50 DEBUG 08:30:50 DEBUG "value": [ 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG "pod": "ds-idrepo-1" 08:30:50 DEBUG 08:30:50 DEBUG }, 08:30:50 DEBUG 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "pod": "ds-idrepo-0" 08:30:50 DEBUG "0" 08:30:50 DEBUG }, 08:30:50 DEBUG ] 08:30:50 DEBUG "value": [ 08:30:50 DEBUG } 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG ] 08:30:50 DEBUG "0" 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "ds-cts-2" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "ds-idrepo-1" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "ds-idrepo-2" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "ds-idrepo-1" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "instance": "10.142.0.17:9100" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "instance": "10.142.0.61:9100" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "6537.366666666667" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "instance": "10.142.0.17:9100" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "instance": "10.142.0.61:9100" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0.00020000000000095497" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "instance": "10.142.0.17:9100" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "instance": "10.142.0.61:9100" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0.2" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "instance": "10.142.0.17:9100" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "instance": "10.142.0.61:9100" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "2772.2000000000003" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "14589767680" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "1.0353782211413392" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "ds-idrepo-0" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0.6840613700772012" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:30:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567850" 08:30:50 INFO [http_cmd]: http status code OK 08:30:50 DEBUG --- status code --- 08:30:50 DEBUG http status code is 200 (expected 200) 08:30:50 DEBUG --- http response --- 08:30:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:30:50 DEBUG ----- output ----- 08:30:50 DEBUG { 08:30:50 DEBUG "status": "success", 08:30:50 DEBUG "data": { 08:30:50 DEBUG "resultType": "vector", 08:30:50 DEBUG "result": [ 08:30:50 DEBUG { 08:30:50 DEBUG "metric": { 08:30:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG [...] 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG 08:30:50 DEBUG "pod": "ds-idrepo-0" 08:30:50 DEBUG }, 08:30:50 DEBUG "value": [ 08:30:50 DEBUG 1684567850, 08:30:50 DEBUG "0.22977693045379632" 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG ] 08:30:50 DEBUG } 08:30:50 DEBUG } 08:31:26 INFO 08:31:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:31:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:31:26 INFO [loop_until]: OK (rc = 0) 08:31:26 DEBUG --- stdout --- 08:31:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 369m 9407Mi am-7849cf7bdb-lxfkl 301m 9343Mi am-7849cf7bdb-td7js 326m 9403Mi ds-cts-0 672m 13849Mi ds-cts-1 1496m 13715Mi ds-cts-2 955m 13639Mi ds-idrepo-0 231m 13661Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 982m 1334Mi 08:31:26 DEBUG --- stderr --- 08:31:26 DEBUG 08:31:38 INFO 08:31:38 INFO [loop_until]: kubectl --namespace=xlou top node 08:31:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:31:38 INFO [loop_until]: OK (rc = 0) 08:31:38 DEBUG --- stdout --- 08:31:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 383m 2% 11563Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4948Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 489m 3% 11678Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 416m 2% 12009Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 155m 0% 3116Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 83m 0% 1364Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 85m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 63m 0% 14643Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1485m 9% 15062Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 283m 1% 14766Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 71m 0% 14574Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 1142m 7% 14970Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 726m 4% 15199Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1085m 6% 3371Mi 5% 08:31:38 DEBUG --- stderr --- 08:31:38 DEBUG 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "ds-idrepo-2" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "ds-idrepo-1" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:31:50 DEBUG "pod": "ds-cts-2" 08:31:50 DEBUG }, 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:31:50 DEBUG "pod": "ds-idrepo-1" 08:31:50 DEBUG }, 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "ds-cts-2" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "ds-idrepo-1" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "ds-idrepo-2" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "ds-idrepo-1" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "ds-idrepo-2" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "ds-idrepo-1" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "instance": "10.142.0.17:9100" 08:31:50 DEBUG "pod": "ds-cts-2" 08:31:50 DEBUG }, 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "instance": "10.142.0.61:9100" 08:31:50 DEBUG "pod": "ds-idrepo-1" 08:31:50 DEBUG }, 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0.0005333333333339852" 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG ] 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567910" 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "instance": "10.142.0.17:9100" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "instance": "10.142.0.61:9100" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "instance": "10.142.0.17:9100" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "instance": "10.142.0.61:9100" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "6667.466666666667" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567910" 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:31:50 DEBUG { 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG { 08:31:50 DEBUG "data": { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "result": [ 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG { 08:31:50 DEBUG "result": [ 08:31:50 DEBUG "metric": { 08:31:50 DEBUG { 08:31:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:31:50 DEBUG "metric": { 08:31:50 DEBUG }, 08:31:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:31:50 DEBUG "value": [ 08:31:50 DEBUG }, 08:31:50 DEBUG 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:31:50 DEBUG 08:31:50 DEBUG }, 08:31:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:31:50 DEBUG "value": [ 08:31:50 DEBUG }, 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG "0" 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG ] 08:31:50 DEBUG "14394183680" 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0.2" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "instance": "10.142.0.17:9100" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "instance": "10.142.0.61:9100" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "3050.2" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567910" 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:31:50 DEBUG }, 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG "value": [ 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG "0" 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG ] 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:31:50 DEBUG } 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG ] 08:31:50 DEBUG { 08:31:50 DEBUG } 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG } 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "1.030785619105157" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:31:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567910" 08:31:50 INFO [http_cmd]: http status code OK 08:31:50 DEBUG --- status code --- 08:31:50 DEBUG http status code is 200 (expected 200) 08:31:50 DEBUG --- http response --- 08:31:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:31:50 DEBUG ----- output ----- 08:31:50 DEBUG { 08:31:50 DEBUG "status": "success", 08:31:50 DEBUG "data": { 08:31:50 DEBUG "resultType": "vector", 08:31:50 DEBUG "result": [ 08:31:50 DEBUG { 08:31:50 DEBUG "metric": { 08:31:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG [...] 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG 08:31:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:31:50 DEBUG }, 08:31:50 DEBUG "value": [ 08:31:50 DEBUG 1684567910, 08:31:50 DEBUG "0" 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG ] 08:31:50 DEBUG } 08:31:50 DEBUG } 08:32:26 INFO 08:32:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:32:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:32:26 INFO [loop_until]: OK (rc = 0) 08:32:26 DEBUG --- stdout --- 08:32:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 362m 9407Mi am-7849cf7bdb-lxfkl 313m 9342Mi am-7849cf7bdb-td7js 319m 9403Mi ds-cts-0 852m 13681Mi ds-cts-1 697m 13750Mi ds-cts-2 675m 13595Mi ds-idrepo-0 232m 13661Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1033m 1577Mi 08:32:26 DEBUG --- stderr --- 08:32:26 DEBUG 08:32:38 INFO 08:32:38 INFO [loop_until]: kubectl --namespace=xlou top node 08:32:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:32:38 INFO [loop_until]: OK (rc = 0) 08:32:38 DEBUG --- stdout --- 08:32:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 385m 2% 11562Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 75m 0% 4946Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 483m 3% 11683Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 407m 2% 12005Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 184m 1% 3086Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 80m 0% 1361Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 82m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 61m 0% 14643Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 748m 4% 15104Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 283m 1% 14767Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14570Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 734m 4% 14928Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 915m 5% 15030Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1079m 6% 3607Mi 6% 08:32:38 DEBUG --- stderr --- 08:32:38 DEBUG 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "ds-idrepo-2" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "ds-idrepo-1" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:32:50 DEBUG "pod": "ds-cts-2" 08:32:50 DEBUG }, 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:32:50 DEBUG "pod": "ds-idrepo-1" 08:32:50 DEBUG }, 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "am-7849cf7bdb-td7js" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "ds-cts-2" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "ds-idrepo-1" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "ds-idrepo-2" 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:32:50 DEBUG }, 08:32:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "ds-idrepo-1" 08:32:50 DEBUG 08:32:50 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:32:50 DEBUG }, 08:32:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "ds-idrepo-2" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "ds-idrepo-1" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "instance": "10.142.0.17:9100" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "instance": "10.142.0.61:9100" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0.0003333333333333262" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567970" 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "instance": "10.142.0.17:9100" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "instance": "10.142.0.61:9100" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0.0005666666666665302" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "ds-cts-2" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "ds-idrepo-1" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "instance": "10.142.0.17:9100" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "instance": "10.142.0.61:9100" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "6747.466666666667" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684567970" 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG { 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:32:50 DEBUG "data": { 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG { 08:32:50 DEBUG "result": [ 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG { 08:32:50 DEBUG "data": { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:32:50 DEBUG "result": [ 08:32:50 DEBUG }, 08:32:50 DEBUG { 08:32:50 DEBUG "value": [ 08:32:50 DEBUG "metric": { 08:32:50 DEBUG 08:32:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:32:50 DEBUG 08:32:50 DEBUG }, 08:32:50 DEBUG 08:32:50 DEBUG "value": [ 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:32:50 DEBUG 08:32:50 DEBUG }, 08:32:50 DEBUG 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:32:50 DEBUG "0" 08:32:50 DEBUG }, 08:32:50 DEBUG ] 08:32:50 DEBUG "value": [ 08:32:50 DEBUG } 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG ] 08:32:50 DEBUG "14336516096" 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0.2" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "instance": "10.142.0.17:9100" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "instance": "10.142.0.61:9100" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "4982.3" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "1.0254026900323172" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:32:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684567970" 08:32:50 INFO [http_cmd]: http status code OK 08:32:50 DEBUG --- status code --- 08:32:50 DEBUG http status code is 200 (expected 200) 08:32:50 DEBUG --- http response --- 08:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:32:50 DEBUG ----- output ----- 08:32:50 DEBUG { 08:32:50 DEBUG "status": "success", 08:32:50 DEBUG "data": { 08:32:50 DEBUG "resultType": "vector", 08:32:50 DEBUG "result": [ 08:32:50 DEBUG { 08:32:50 DEBUG "metric": { 08:32:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG [...] 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG 08:32:50 DEBUG "pod": "am-7849cf7bdb-td7js" 08:32:50 DEBUG }, 08:32:50 DEBUG "value": [ 08:32:50 DEBUG 1684567970, 08:32:50 DEBUG "0" 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG ] 08:32:50 DEBUG } 08:32:50 DEBUG } 08:33:26 INFO 08:33:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:33:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:33:26 INFO [loop_until]: OK (rc = 0) 08:33:26 DEBUG --- stdout --- 08:33:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 589m 9406Mi am-7849cf7bdb-lxfkl 318m 9342Mi am-7849cf7bdb-td7js 322m 9403Mi ds-cts-0 873m 13725Mi ds-cts-1 1309m 13672Mi ds-cts-2 664m 13630Mi ds-idrepo-0 231m 13661Mi ds-idrepo-1 13m 13554Mi ds-idrepo-2 16m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1014m 1820Mi 08:33:26 DEBUG --- stderr --- 08:33:26 DEBUG 08:33:38 INFO 08:33:38 INFO [loop_until]: kubectl --namespace=xlou top node 08:33:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:33:38 INFO [loop_until]: OK (rc = 0) 08:33:38 DEBUG --- stdout --- 08:33:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 404m 2% 11559Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4950Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 746m 4% 11683Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 417m 2% 12009Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 151m 0% 3120Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 85m 0% 1360Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 83m 0% 5297Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 62m 0% 14645Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1397m 8% 15019Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 283m 1% 14769Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14574Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 720m 4% 14969Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 939m 5% 15074Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1077m 6% 3842Mi 6% 08:33:38 DEBUG --- stderr --- 08:33:38 DEBUG 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "ds-idrepo-2" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "ds-idrepo-1" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "0" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG { 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "data": { 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG "result": [ 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG { 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "ds-cts-2" 08:33:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:33:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:33:50 DEBUG }, 08:33:50 DEBUG }, 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG "value": [ 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG [...] 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "ds-idrepo-1" 08:33:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:33:50 DEBUG "pod": "ds-idrepo-1" 08:33:50 DEBUG }, 08:33:50 DEBUG }, 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG "value": [ 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "0" 08:33:50 DEBUG "0" 08:33:50 DEBUG "0" 08:33:50 DEBUG ] 08:33:50 DEBUG ] 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG ] 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "ds-cts-2" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "ds-idrepo-1" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "0" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:33:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:33:50 DEBUG }, 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:33:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:33:50 DEBUG }, 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "0" 08:33:50 DEBUG "0" 08:33:50 DEBUG ] 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568030" 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568030" 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG { 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG "data": { 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG { 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG "metric": { 08:33:50 DEBUG { 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG "pod": "ds-idrepo-2" 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG }, 08:33:50 DEBUG "data": { 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:33:50 DEBUG "value": [ 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG ----- output ----- 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "data": { 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [...] 08:33:50 DEBUG "instance": "10.142.0.17:9100" 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:33:50 DEBUG 08:33:50 DEBUG }, 08:33:50 DEBUG "result": [ 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG 08:33:50 DEBUG "value": [ 08:33:50 DEBUG { 08:33:50 DEBUG { 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "pod": "ds-idrepo-1" 08:33:50 DEBUG 08:33:50 DEBUG "instance": "10.142.0.17:9100" 08:33:50 DEBUG "data": { 08:33:50 DEBUG }, 08:33:50 DEBUG 08:33:50 DEBUG }, 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "value": [ 08:33:50 DEBUG [...] 08:33:50 DEBUG "value": [ 08:33:50 DEBUG "result": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG { 08:33:50 DEBUG "0" 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "metric": { 08:33:50 DEBUG ] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:33:50 DEBUG } 08:33:50 DEBUG "instance": "10.142.0.61:9100" 08:33:50 DEBUG [...] 08:33:50 DEBUG }, 08:33:50 DEBUG ] 08:33:50 DEBUG }, 08:33:50 DEBUG 08:33:50 DEBUG "value": [ 08:33:50 DEBUG } 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG } 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "0.0004999999999995453" 08:33:50 DEBUG "instance": "10.142.0.61:9100" 08:33:50 DEBUG 08:33:50 DEBUG ] 08:33:50 DEBUG }, 08:33:50 DEBUG [...] 08:33:50 DEBUG } 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG ] 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG 08:33:50 DEBUG } 08:33:50 DEBUG "0" 08:33:50 DEBUG 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG "pod": "ds-idrepo-2" 08:33:50 DEBUG } 08:33:50 DEBUG }, 08:33:50 DEBUG ] 08:33:50 DEBUG "value": [ 08:33:50 DEBUG } 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG } 08:33:50 DEBUG "0" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "ds-idrepo-2" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "ds-idrepo-1" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "0" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "ds-cts-2" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "ds-idrepo-1" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "0" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "instance": "10.142.0.17:9100" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "instance": "10.142.0.61:9100" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "6682.966666666667" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 DEBUG 1684568030, 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG "0" 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG ] 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG } 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG ] 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:33:50 DEBUG } 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG } 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "14393503744" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "0.2" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "instance": "10.142.0.17:9100" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "instance": "10.142.0.61:9100" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "2868.0333333333333" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "ds-idrepo-2" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "0.6113115263185135" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "1.0335376658960838" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:33:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568030" 08:33:50 INFO [http_cmd]: http status code OK 08:33:50 DEBUG --- status code --- 08:33:50 DEBUG http status code is 200 (expected 200) 08:33:50 DEBUG --- http response --- 08:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:33:50 DEBUG ----- output ----- 08:33:50 DEBUG { 08:33:50 DEBUG "status": "success", 08:33:50 DEBUG "data": { 08:33:50 DEBUG "resultType": "vector", 08:33:50 DEBUG "result": [ 08:33:50 DEBUG { 08:33:50 DEBUG "metric": { 08:33:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG [...] 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG 08:33:50 DEBUG "pod": "ds-idrepo-1" 08:33:50 DEBUG }, 08:33:50 DEBUG "value": [ 08:33:50 DEBUG 1684568030, 08:33:50 DEBUG "0" 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG ] 08:33:50 DEBUG } 08:33:50 DEBUG } 08:34:26 INFO 08:34:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:34:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:34:26 INFO [loop_until]: OK (rc = 0) 08:34:26 DEBUG --- stdout --- 08:34:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 351m 9406Mi am-7849cf7bdb-lxfkl 300m 9342Mi am-7849cf7bdb-td7js 302m 9403Mi ds-cts-0 713m 13711Mi ds-cts-1 956m 13724Mi ds-cts-2 1118m 13727Mi ds-idrepo-0 224m 13661Mi ds-idrepo-1 13m 13554Mi ds-idrepo-2 16m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 999m 2067Mi 08:34:26 DEBUG --- stderr --- 08:34:26 DEBUG 08:34:38 INFO 08:34:38 INFO [loop_until]: kubectl --namespace=xlou top node 08:34:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:34:38 INFO [loop_until]: OK (rc = 0) 08:34:38 DEBUG --- stdout --- 08:34:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 384m 2% 11562Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4948Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 462m 2% 11687Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 414m 2% 12008Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 149m 0% 3133Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 85m 0% 1359Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 88m 0% 5300Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 60m 0% 14644Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1012m 6% 15070Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 274m 1% 14766Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14571Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 1356m 8% 15056Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 702m 4% 15067Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1086m 6% 4077Mi 6% 08:34:38 DEBUG --- stderr --- 08:34:38 DEBUG 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "pod": "ds-idrepo-2" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "ds-idrepo-1" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG { 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "data": { 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG "result": [ 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG { 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "pod": "ds-cts-2" 08:34:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:34:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:34:50 DEBUG }, 08:34:50 DEBUG }, 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG "value": [ 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG [...] 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "ds-idrepo-1" 08:34:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:34:50 DEBUG "pod": "ds-idrepo-2" 08:34:50 DEBUG }, 08:34:50 DEBUG }, 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG "value": [ 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0" 08:34:50 DEBUG "0" 08:34:50 DEBUG "451359388.4242626" 08:34:50 DEBUG ] 08:34:50 DEBUG ] 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG ] 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "pod": "ds-cts-2" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "ds-idrepo-1" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "pod": "ds-idrepo-2" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "ds-idrepo-1" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:34:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:34:50 DEBUG }, 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:34:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:34:50 DEBUG }, 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0" 08:34:50 DEBUG "0" 08:34:50 DEBUG ] 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568090" 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568090" 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG { 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG { 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG "data": { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "data": { 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:34:50 DEBUG "result": [ 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "pod": "ds-cts-2" 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "data": { 08:34:50 DEBUG }, 08:34:50 DEBUG "instance": "10.142.0.17:9100" 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "value": [ 08:34:50 DEBUG }, 08:34:50 DEBUG "result": [ 08:34:50 DEBUG 08:34:50 DEBUG "value": [ 08:34:50 DEBUG { 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "metric": { 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "instance": "10.142.0.17:9100" 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG }, 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "ds-idrepo-1" 08:34:50 DEBUG { 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG }, 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "instance": "10.142.0.61:9100" 08:34:50 DEBUG [...] 08:34:50 DEBUG "value": [ 08:34:50 DEBUG "data": { 08:34:50 DEBUG }, 08:34:50 DEBUG 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG "0" 08:34:50 DEBUG "result": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG 08:34:50 DEBUG ] 08:34:50 DEBUG { 08:34:50 DEBUG "0.00036666666666747006" 08:34:50 DEBUG "instance": "10.142.0.61:9100" 08:34:50 DEBUG } 08:34:50 DEBUG "metric": { 08:34:50 DEBUG ] 08:34:50 DEBUG }, 08:34:50 DEBUG ] 08:34:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:34:50 DEBUG } 08:34:50 DEBUG "value": [ 08:34:50 DEBUG } 08:34:50 DEBUG }, 08:34:50 DEBUG ] 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG } 08:34:50 DEBUG "value": [ 08:34:50 DEBUG } 08:34:50 DEBUG "0.00033333333333397756" 08:34:50 DEBUG 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG 08:34:50 DEBUG } 08:34:50 DEBUG 08:34:50 DEBUG ] 08:34:50 DEBUG [...] 08:34:50 DEBUG } 08:34:50 DEBUG 08:34:50 DEBUG } 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "ds-idrepo-2" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "pod": "ds-idrepo-2" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "ds-idrepo-1" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "instance": "10.142.0.17:9100" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "instance": "10.142.0.61:9100" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "6582.966666666667" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568090" 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG { 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG "data": { 08:34:50 DEBUG { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG "data": { 08:34:50 DEBUG { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "result": [ 08:34:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:34:50 DEBUG { 08:34:50 DEBUG }, 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "value": [ 08:34:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:34:50 DEBUG 08:34:50 DEBUG }, 08:34:50 DEBUG 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:34:50 DEBUG 08:34:50 DEBUG }, 08:34:50 DEBUG 08:34:50 DEBUG "value": [ 08:34:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG }, 08:34:50 DEBUG "14472298496" 08:34:50 DEBUG "value": [ 08:34:50 DEBUG ] 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG } 08:34:50 DEBUG "0" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG ] 08:34:50 DEBUG ] 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG } 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:34:50 DEBUG } 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0.2" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG ] 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG } 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG } 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "instance": "10.142.0.17:9100" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "instance": "10.142.0.61:9100" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "2986.0666666666666" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "ds-idrepo-2" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0.9057697533287038" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "1.0316053189278425" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:34:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568090" 08:34:50 INFO [http_cmd]: http status code OK 08:34:50 DEBUG --- status code --- 08:34:50 DEBUG http status code is 200 (expected 200) 08:34:50 DEBUG --- http response --- 08:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:34:50 DEBUG ----- output ----- 08:34:50 DEBUG { 08:34:50 DEBUG "status": "success", 08:34:50 DEBUG "data": { 08:34:50 DEBUG "resultType": "vector", 08:34:50 DEBUG "result": [ 08:34:50 DEBUG { 08:34:50 DEBUG "metric": { 08:34:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG [...] 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG 08:34:50 DEBUG "pod": "ds-idrepo-2" 08:34:50 DEBUG }, 08:34:50 DEBUG "value": [ 08:34:50 DEBUG 1684568090, 08:34:50 DEBUG "0.014152313909600306" 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG ] 08:34:50 DEBUG } 08:34:50 DEBUG } 08:35:26 INFO 08:35:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:35:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:35:26 INFO [loop_until]: OK (rc = 0) 08:35:26 DEBUG --- stdout --- 08:35:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 365m 9403Mi am-7849cf7bdb-lxfkl 308m 9342Mi am-7849cf7bdb-td7js 317m 9403Mi ds-cts-0 892m 13757Mi ds-cts-1 1445m 13565Mi ds-cts-2 655m 13779Mi ds-idrepo-0 237m 13661Mi ds-idrepo-1 14m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 4m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1005m 2304Mi 08:35:26 DEBUG --- stderr --- 08:35:26 DEBUG 08:35:38 INFO 08:35:38 INFO [loop_until]: kubectl --namespace=xlou top node 08:35:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:35:38 INFO [loop_until]: OK (rc = 0) 08:35:38 DEBUG --- stdout --- 08:35:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 375m 2% 11564Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 70m 0% 4951Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 474m 2% 11676Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 406m 2% 12008Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 155m 0% 3132Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 85m 0% 1359Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 89m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 63m 0% 14641Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1617m 10% 14916Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 277m 1% 14768Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14576Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 793m 4% 15109Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 889m 5% 15114Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1079m 6% 4313Mi 7% 08:35:38 DEBUG --- stderr --- 08:35:38 DEBUG 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "pod": "ds-idrepo-2" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "pod": "ds-idrepo-1" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "0" 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG { 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "data": { 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG "result": [ 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG { 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:35:50 DEBUG "pod": "ds-cts-2" 08:35:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:35:50 DEBUG }, 08:35:50 DEBUG }, 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG [...] 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:35:50 DEBUG "pod": "ds-idrepo-1" 08:35:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:35:50 DEBUG }, 08:35:50 DEBUG }, 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "9007.829622458858" 08:35:50 DEBUG "0" 08:35:50 DEBUG "0" 08:35:50 DEBUG ] 08:35:50 DEBUG ] 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG ] 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "pod": "ds-idrepo-2" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "pod": "ds-idrepo-1" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "0" 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "pod": "ds-cts-2" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "pod": "ds-idrepo-1" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "0" 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG --- http response --- 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG { 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "data": { 08:35:50 DEBUG ----- output ----- 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG "result": [ 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG { 08:35:50 DEBUG "data": { 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "instance": "10.142.0.17:9100" 08:35:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:35:50 DEBUG "result": [ 08:35:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:35:50 DEBUG }, 08:35:50 DEBUG }, 08:35:50 DEBUG { 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "instance": "10.142.0.17:9100" 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG }, 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG "instance": "10.142.0.61:9100" 08:35:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:35:50 DEBUG 08:35:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:35:50 DEBUG }, 08:35:50 DEBUG }, 08:35:50 DEBUG 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "instance": "10.142.0.61:9100" 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "0.0003333333333330302" 08:35:50 DEBUG "0" 08:35:50 DEBUG }, 08:35:50 DEBUG "0" 08:35:50 DEBUG ] 08:35:50 DEBUG ] 08:35:50 DEBUG "value": [ 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG ] 08:35:50 DEBUG "0" 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "pod": "ds-cts-2" 08:35:50 DEBUG "pod": "ds-idrepo-2" 08:35:50 DEBUG }, 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "pod": "ds-idrepo-1" 08:35:50 DEBUG "pod": "ds-idrepo-1" 08:35:50 DEBUG }, 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "0" 08:35:50 DEBUG "0" 08:35:50 DEBUG ] 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568150" 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "0" 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "instance": "10.142.0.17:9100" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "instance": "10.142.0.61:9100" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "6766.633333333333" 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568150" 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG "data": { 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG "result": [ 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG { 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "data": { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:35:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:35:50 DEBUG "resultType": "vector", 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG }, 08:35:50 DEBUG "result": [ 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG }, 08:35:50 DEBUG { 08:35:50 DEBUG "value": [ 08:35:50 DEBUG { 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG "value": [ 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG 08:35:50 DEBUG "metric": { 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG 08:35:50 DEBUG "data": { 08:35:50 DEBUG 08:35:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG 08:35:50 DEBUG }, 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG 08:35:50 DEBUG "result": [ 08:35:50 DEBUG [...] 08:35:50 DEBUG "value": [ 08:35:50 DEBUG { 08:35:50 DEBUG [...] 08:35:50 DEBUG { 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG 08:35:50 DEBUG "metric": { 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "data": { 08:35:50 DEBUG 08:35:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG 08:35:50 DEBUG }, 08:35:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:35:50 DEBUG [...] 08:35:50 DEBUG "result": [ 08:35:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:35:50 DEBUG "value": [ 08:35:50 DEBUG }, 08:35:50 DEBUG 08:35:50 DEBUG { 08:35:50 DEBUG }, 08:35:50 DEBUG 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG 08:35:50 DEBUG "instance": "10.142.0.17:9100" 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG 08:35:50 DEBUG "0" 08:35:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:35:50 DEBUG }, 08:35:50 DEBUG "0" 08:35:50 DEBUG [...] 08:35:50 DEBUG ] 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG ] 08:35:50 DEBUG 08:35:50 DEBUG } 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG } 08:35:50 DEBUG 08:35:50 DEBUG ] 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG 08:35:50 DEBUG ] 08:35:50 DEBUG 08:35:50 DEBUG } 08:35:50 DEBUG "14540754944" 08:35:50 DEBUG 08:35:50 DEBUG } 08:35:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG [...] 08:35:50 DEBUG } 08:35:50 DEBUG }, 08:35:50 DEBUG } 08:35:50 DEBUG 08:35:50 DEBUG "value": [ 08:35:50 DEBUG ] 08:35:50 DEBUG 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG } 08:35:50 DEBUG 08:35:50 DEBUG "0.2" 08:35:50 DEBUG } 08:35:50 DEBUG "instance": "10.142.0.61:9100" 08:35:50 DEBUG ] 08:35:50 DEBUG }, 08:35:50 DEBUG } 08:35:50 DEBUG "value": [ 08:35:50 DEBUG ] 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG } 08:35:50 DEBUG "3247.2000000000003" 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568150" 08:35:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568150" 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "0.43883616666708924" 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:35:50 INFO [http_cmd]: http status code OK 08:35:50 DEBUG --- status code --- 08:35:50 DEBUG http status code is 200 (expected 200) 08:35:50 DEBUG --- http response --- 08:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:35:50 DEBUG ----- output ----- 08:35:50 DEBUG { 08:35:50 DEBUG "status": "success", 08:35:50 DEBUG "data": { 08:35:50 DEBUG "resultType": "vector", 08:35:50 DEBUG "result": [ 08:35:50 DEBUG { 08:35:50 DEBUG "metric": { 08:35:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG [...] 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG 08:35:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:35:50 DEBUG }, 08:35:50 DEBUG "value": [ 08:35:50 DEBUG 1684568150, 08:35:50 DEBUG "1.0287978308661292" 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG ] 08:35:50 DEBUG } 08:35:50 DEBUG } 08:36:26 INFO 08:36:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:36:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:36:26 INFO [loop_until]: OK (rc = 0) 08:36:26 DEBUG --- stdout --- 08:36:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 371m 9403Mi am-7849cf7bdb-lxfkl 301m 9342Mi am-7849cf7bdb-td7js 324m 9403Mi ds-cts-0 928m 13805Mi ds-cts-1 697m 13628Mi ds-cts-2 1417m 13746Mi ds-idrepo-0 241m 13661Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 8m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1029m 2537Mi 08:36:26 DEBUG --- stderr --- 08:36:26 DEBUG 08:36:38 INFO 08:36:38 INFO [loop_until]: kubectl --namespace=xlou top node 08:36:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:36:38 INFO [loop_until]: OK (rc = 0) 08:36:38 DEBUG --- stdout --- 08:36:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 411m 2% 11561Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 75m 0% 4950Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 484m 3% 11679Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 405m 2% 12007Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 151m 0% 3131Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 87m 0% 1360Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 90m 0% 5303Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 61m 0% 14643Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 730m 4% 14982Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 287m 1% 14769Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14577Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 1586m 9% 15108Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 933m 5% 15161Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1080m 6% 4547Mi 7% 08:36:38 DEBUG --- stderr --- 08:36:38 DEBUG 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "ds-idrepo-2" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-idrepo-1" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-idrepo-2" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "5404.613942466851" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "ds-cts-2" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-idrepo-1" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "ds-idrepo-2" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-idrepo-1" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "ds-cts-2" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-idrepo-1" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568210" 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568210" 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG { 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG "data": { 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "data": { 08:36:50 DEBUG "pod": "ds-idrepo-2" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG }, 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG "result": [ 08:36:50 DEBUG "value": [ 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG { 08:36:50 DEBUG 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG "metric": { 08:36:50 DEBUG 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:36:50 DEBUG "instance": "10.142.0.17:9100" 08:36:50 DEBUG 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG }, 08:36:50 DEBUG [...] 08:36:50 DEBUG { 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "data": { 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-idrepo-1" 08:36:50 DEBUG "result": [ 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG [...] 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG }, 08:36:50 DEBUG { 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG "value": [ 08:36:50 DEBUG "metric": { 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "instance": "10.142.0.17:9100" 08:36:50 DEBUG --- http response --- 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG "0" 08:36:50 DEBUG }, 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG "instance": "10.142.0.61:9100" 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:36:50 DEBUG ] 08:36:50 DEBUG "value": [ 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG }, 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG } 08:36:50 DEBUG 08:36:50 DEBUG { 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG "value": [ 08:36:50 DEBUG { 08:36:50 DEBUG ] 08:36:50 DEBUG 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG } 08:36:50 DEBUG 08:36:50 DEBUG "data": { 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG "6691.733333333334" 08:36:50 DEBUG "data": { 08:36:50 DEBUG } 08:36:50 DEBUG [...] 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG { 08:36:50 DEBUG ] 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG 08:36:50 DEBUG "result": [ 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG } 08:36:50 DEBUG "result": [ 08:36:50 DEBUG 08:36:50 DEBUG { 08:36:50 DEBUG "data": { 08:36:50 DEBUG ] 08:36:50 DEBUG { 08:36:50 DEBUG 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG } 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "instance": "10.142.0.61:9100" 08:36:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:36:50 DEBUG "result": [ 08:36:50 DEBUG } 08:36:50 DEBUG "instance": "10.142.0.17:9100" 08:36:50 DEBUG }, 08:36:50 DEBUG }, 08:36:50 DEBUG { 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG "value": [ 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-cts-2" 08:36:50 DEBUG 08:36:50 DEBUG "0.0004333333333325603" 08:36:50 DEBUG 08:36:50 DEBUG }, 08:36:50 DEBUG 08:36:50 DEBUG ] 08:36:50 DEBUG 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG } 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG ] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG } 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG } 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG "instance": "10.142.0.61:9100" 08:36:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:36:50 DEBUG 08:36:50 DEBUG }, 08:36:50 DEBUG }, 08:36:50 DEBUG 08:36:50 DEBUG "value": [ 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "pod": "ds-idrepo-1" 08:36:50 DEBUG "0" 08:36:50 DEBUG "0" 08:36:50 DEBUG }, 08:36:50 DEBUG ] 08:36:50 DEBUG ] 08:36:50 DEBUG "value": [ 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG ] 08:36:50 DEBUG ] 08:36:50 DEBUG "0" 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-idrepo-2" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0.2" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "14514401280" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "instance": "10.142.0.17:9100" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "instance": "10.142.0.61:9100" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "3229.1666666666665" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG }, 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:36:50 DEBUG "value": [ 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG { 08:36:50 DEBUG "0" 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG ] 08:36:50 DEBUG "data": { 08:36:50 DEBUG } 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG ] 08:36:50 DEBUG "result": [ 08:36:50 DEBUG } 08:36:50 DEBUG { 08:36:50 DEBUG } 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-idrepo-2" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0.590296290488387" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568210" 08:36:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568210" 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "pod": "ds-idrepo-2" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "0.010145989549931279" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:36:50 INFO [http_cmd]: http status code OK 08:36:50 DEBUG --- status code --- 08:36:50 DEBUG http status code is 200 (expected 200) 08:36:50 DEBUG --- http response --- 08:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:36:50 DEBUG ----- output ----- 08:36:50 DEBUG { 08:36:50 DEBUG "status": "success", 08:36:50 DEBUG "data": { 08:36:50 DEBUG "resultType": "vector", 08:36:50 DEBUG "result": [ 08:36:50 DEBUG { 08:36:50 DEBUG "metric": { 08:36:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG [...] 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG 08:36:50 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:36:50 DEBUG }, 08:36:50 DEBUG "value": [ 08:36:50 DEBUG 1684568210, 08:36:50 DEBUG "1.0319790734889236" 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG ] 08:36:50 DEBUG } 08:36:50 DEBUG } 08:37:26 INFO 08:37:26 INFO [loop_until]: kubectl --namespace=xlou top pods 08:37:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:37:27 INFO [loop_until]: OK (rc = 0) 08:37:27 DEBUG --- stdout --- 08:37:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 377m 9403Mi am-7849cf7bdb-lxfkl 311m 9342Mi am-7849cf7bdb-td7js 311m 9403Mi ds-cts-0 625m 13572Mi ds-cts-1 694m 13715Mi ds-cts-2 667m 13777Mi ds-idrepo-0 236m 13662Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1027m 2757Mi 08:37:27 DEBUG --- stderr --- 08:37:27 DEBUG 08:37:39 INFO 08:37:39 INFO [loop_until]: kubectl --namespace=xlou top node 08:37:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:37:39 INFO [loop_until]: OK (rc = 0) 08:37:39 DEBUG --- stdout --- 08:37:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 392m 2% 11561Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4952Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 491m 3% 11679Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 423m 2% 12006Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 170m 1% 3065Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 85m 0% 1359Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 87m 0% 5301Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 63m 0% 14642Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 738m 4% 15061Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 286m 1% 14772Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14577Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 731m 4% 15134Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 671m 4% 14933Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1044m 6% 4781Mi 8% 08:37:39 DEBUG --- stderr --- 08:37:39 DEBUG 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "ds-idrepo-2" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "ds-idrepo-1" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "3710.783140887021" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "ds-cts-2" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "ds-idrepo-1" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "ds-idrepo-2" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "ds-idrepo-1" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "ds-cts-2" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "ds-idrepo-1" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "ds-idrepo-2" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "ds-idrepo-1" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568270" 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568270" 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568270" 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "instance": "10.142.0.17:9100" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "instance": "10.142.0.61:9100" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG }, 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG "value": [ 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG "6801.633333333333" 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:50 DEBUG ] 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG } 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG { 08:37:50 DEBUG ] 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG } 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG "data": { 08:37:50 DEBUG } 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG ----- output ----- 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG { 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG "metric": { 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG "data": { 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG "instance": "10.142.0.17:9100" 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:50 DEBUG }, 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG "result": [ 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG "value": [ 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:50 DEBUG { 08:37:50 DEBUG { 08:37:50 DEBUG 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG { 08:37:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:37:50 DEBUG "data": { 08:37:50 DEBUG 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG }, 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG [...] 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG "data": { 08:37:50 DEBUG "value": [ 08:37:50 DEBUG "result": [ 08:37:50 DEBUG 08:37:50 DEBUG { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG 08:37:50 DEBUG { 08:37:50 DEBUG 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG 08:37:50 DEBUG "metric": { 08:37:50 DEBUG 08:37:50 DEBUG "data": { 08:37:50 DEBUG { 08:37:50 DEBUG 08:37:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:37:50 DEBUG "instance": "10.142.0.61:9100" 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "metric": { 08:37:50 DEBUG [...] 08:37:50 DEBUG }, 08:37:50 DEBUG }, 08:37:50 DEBUG "result": [ 08:37:50 DEBUG "instance": "10.142.0.17:9100" 08:37:50 DEBUG 08:37:50 DEBUG "value": [ 08:37:50 DEBUG "value": [ 08:37:50 DEBUG { 08:37:50 DEBUG }, 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "0.00036666666666747006" 08:37:50 DEBUG "pod": "ds-cts-2" 08:37:50 DEBUG 08:37:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:37:50 DEBUG 08:37:50 DEBUG ] 08:37:50 DEBUG }, 08:37:50 DEBUG 08:37:50 DEBUG }, 08:37:50 DEBUG [...] 08:37:50 DEBUG } 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG ] 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG 08:37:50 DEBUG } 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "0" 08:37:50 DEBUG 08:37:50 DEBUG } 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG ] 08:37:50 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG } 08:37:50 DEBUG }, 08:37:50 DEBUG 08:37:50 DEBUG "instance": "10.142.0.61:9100" 08:37:50 DEBUG ] 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG }, 08:37:50 DEBUG } 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG 08:37:50 DEBUG "value": [ 08:37:50 DEBUG } 08:37:50 DEBUG "0" 08:37:50 DEBUG "pod": "ds-idrepo-1" 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG ] 08:37:50 DEBUG }, 08:37:50 DEBUG "0" 08:37:50 DEBUG } 08:37:50 DEBUG "value": [ 08:37:50 DEBUG ] 08:37:50 DEBUG ] 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0.2" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "14548774912" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "instance": "10.142.0.17:9100" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "instance": "10.142.0.61:9100" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "5061.5" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568270" 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:50 INFO [http_cmd]: http status code OK 08:37:50 DEBUG --- status code --- 08:37:50 DEBUG http status code is 200 (expected 200) 08:37:50 DEBUG --- http response --- 08:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:50 DEBUG ----- output ----- 08:37:50 DEBUG { 08:37:50 DEBUG "status": "success", 08:37:50 DEBUG "data": { 08:37:50 DEBUG "resultType": "vector", 08:37:50 DEBUG "result": [ 08:37:50 DEBUG { 08:37:50 DEBUG "metric": { 08:37:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG [...] 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG 08:37:50 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:37:50 DEBUG }, 08:37:50 DEBUG "value": [ 08:37:50 DEBUG 1684568270, 08:37:50 DEBUG "0" 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG ] 08:37:50 DEBUG } 08:37:50 DEBUG } 08:37:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568271" 08:37:51 INFO [http_cmd]: http status code OK 08:37:51 DEBUG --- status code --- 08:37:51 DEBUG http status code is 200 (expected 200) 08:37:51 DEBUG --- http response --- 08:37:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:51 DEBUG ----- output ----- 08:37:51 DEBUG { 08:37:51 DEBUG "status": "success", 08:37:51 DEBUG "data": { 08:37:51 DEBUG "resultType": "vector", 08:37:51 DEBUG "result": [ 08:37:51 DEBUG { 08:37:51 DEBUG "metric": { 08:37:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:37:51 DEBUG }, 08:37:51 DEBUG "value": [ 08:37:51 DEBUG 08:37:51 DEBUG 08:37:51 DEBUG 08:37:51 DEBUG [...] 08:37:51 DEBUG 08:37:51 DEBUG 08:37:51 DEBUG 08:37:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:37:51 DEBUG }, 08:37:51 DEBUG "value": [ 08:37:51 DEBUG 1684568271, 08:37:51 DEBUG "1.021717612961511" 08:37:51 DEBUG ] 08:37:51 DEBUG } 08:37:51 DEBUG ] 08:37:51 DEBUG } 08:37:51 DEBUG } 08:37:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568271" 08:37:51 INFO [http_cmd]: http status code OK 08:37:51 DEBUG --- status code --- 08:37:51 DEBUG http status code is 200 (expected 200) 08:37:51 DEBUG --- http response --- 08:37:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:37:51 DEBUG ----- output ----- 08:37:51 DEBUG { 08:37:51 DEBUG "status": "success", 08:37:51 DEBUG "data": { 08:37:51 DEBUG "resultType": "vector", 08:37:51 DEBUG "result": [ 08:37:51 DEBUG { 08:37:51 DEBUG "metric": { 08:37:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:37:51 DEBUG }, 08:37:51 DEBUG "value": [ 08:37:51 DEBUG 08:37:51 DEBUG 08:37:51 DEBUG 08:37:51 DEBUG [...] 08:37:51 DEBUG 08:37:51 DEBUG 08:37:51 DEBUG 08:37:51 DEBUG "pod": "ds-cts-0" 08:37:51 DEBUG }, 08:37:51 DEBUG "value": [ 08:37:51 DEBUG 1684568271, 08:37:51 DEBUG "0.7774996749954974" 08:37:51 DEBUG ] 08:37:51 DEBUG } 08:37:51 DEBUG ] 08:37:51 DEBUG } 08:37:51 DEBUG } 08:38:27 INFO 08:38:27 INFO [loop_until]: kubectl --namespace=xlou top pods 08:38:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:38:27 INFO [loop_until]: OK (rc = 0) 08:38:27 DEBUG --- stdout --- 08:38:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 360m 9403Mi am-7849cf7bdb-lxfkl 313m 9342Mi am-7849cf7bdb-td7js 308m 9402Mi ds-cts-0 858m 13609Mi ds-cts-1 822m 13765Mi ds-cts-2 630m 13806Mi ds-idrepo-0 230m 13661Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 996m 2971Mi 08:38:27 DEBUG --- stderr --- 08:38:27 DEBUG 08:38:39 INFO 08:38:39 INFO [loop_until]: kubectl --namespace=xlou top node 08:38:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:38:39 INFO [loop_until]: OK (rc = 0) 08:38:39 DEBUG --- stdout --- 08:38:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 402m 2% 11561Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4946Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 473m 2% 11680Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 401m 2% 12009Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 158m 0% 3122Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 81m 0% 1363Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 87m 0% 5302Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 61m 0% 14645Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1172m 7% 15128Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 285m 1% 14769Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14577Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 838m 5% 15162Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 918m 5% 14977Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1078m 6% 5014Mi 8% 08:38:39 DEBUG --- stderr --- 08:38:39 DEBUG 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "pod": "ds-idrepo-2" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "ds-idrepo-1" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "0" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "0" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "am-7849cf7bdb-td7js" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "251685225.7885862" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "pod": "ds-cts-2" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "ds-idrepo-1" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "0" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "pod": "ds-idrepo-2" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "ds-idrepo-1" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "0" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "pod": "ds-cts-2" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "ds-idrepo-1" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "0" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "0" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "pod": "ds-idrepo-2" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "ds-idrepo-1" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "0" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568330" 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568330" 08:38:50 DEBUG --- http response --- 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "instance": "10.142.0.17:9100" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "instance": "10.142.0.61:9100" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "6515.966666666666" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG { 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG "data": { 08:38:50 DEBUG { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG "data": { 08:38:50 DEBUG { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "result": [ 08:38:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:38:50 DEBUG { 08:38:50 DEBUG }, 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "value": [ 08:38:50 DEBUG "instance": "10.142.0.17:9100" 08:38:50 DEBUG 08:38:50 DEBUG }, 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG 08:38:50 DEBUG "value": [ 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG { 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:38:50 DEBUG 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG }, 08:38:50 DEBUG 08:38:50 DEBUG "data": { 08:38:50 DEBUG { 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:38:50 DEBUG "value": [ 08:38:50 DEBUG "instance": "10.142.0.61:9100" 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG }, 08:38:50 DEBUG "result": [ 08:38:50 DEBUG "data": { 08:38:50 DEBUG { 08:38:50 DEBUG "0" 08:38:50 DEBUG "value": [ 08:38:50 DEBUG { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG ] 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "result": [ 08:38:50 DEBUG "data": { 08:38:50 DEBUG } 08:38:50 DEBUG "0.0001666666666665151" 08:38:50 DEBUG "pod": "ds-cts-2" 08:38:50 DEBUG { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG ] 08:38:50 DEBUG ] 08:38:50 DEBUG }, 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "result": [ 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 DEBUG "value": [ 08:38:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:38:50 DEBUG { 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG 08:38:50 DEBUG }, 08:38:50 DEBUG "metric": { 08:38:50 DEBUG } 08:38:50 DEBUG 08:38:50 DEBUG "value": [ 08:38:50 DEBUG "instance": "10.142.0.17:9100" 08:38:50 DEBUG } 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG }, 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "ds-idrepo-1" 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG }, 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "value": [ 08:38:50 DEBUG "pod": "am-7849cf7bdb-td7js" 08:38:50 DEBUG 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG }, 08:38:50 DEBUG 08:38:50 DEBUG "0" 08:38:50 DEBUG "value": [ 08:38:50 DEBUG "instance": "10.142.0.61:9100" 08:38:50 DEBUG ] 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG }, 08:38:50 DEBUG } 08:38:50 DEBUG "0" 08:38:50 DEBUG "value": [ 08:38:50 DEBUG ] 08:38:50 DEBUG ] 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 DEBUG "0" 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "0.2" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "14568521728" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568330" 08:38:50 INFO [http_cmd]: http status code OK 08:38:50 DEBUG --- status code --- 08:38:50 DEBUG http status code is 200 (expected 200) 08:38:50 DEBUG --- http response --- 08:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:38:50 DEBUG ----- output ----- 08:38:50 DEBUG { 08:38:50 DEBUG "status": "success", 08:38:50 DEBUG "data": { 08:38:50 DEBUG "resultType": "vector", 08:38:50 DEBUG "result": [ 08:38:50 DEBUG { 08:38:50 DEBUG "metric": { 08:38:50 DEBUG "instance": "10.142.0.17:9100" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG [...] 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG 08:38:50 DEBUG "instance": "10.142.0.61:9100" 08:38:50 DEBUG }, 08:38:50 DEBUG "value": [ 08:38:50 DEBUG 1684568330, 08:38:50 DEBUG "2761.6" 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG ] 08:38:50 DEBUG } 08:38:50 DEBUG } 08:38:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568331" 08:38:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568331" 08:38:51 INFO [http_cmd]: http status code OK 08:38:51 DEBUG --- status code --- 08:38:51 DEBUG http status code is 200 (expected 200) 08:38:51 DEBUG --- http response --- 08:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:38:51 DEBUG ----- output ----- 08:38:51 DEBUG { 08:38:51 DEBUG "status": "success", 08:38:51 DEBUG "data": { 08:38:51 DEBUG "resultType": "vector", 08:38:51 DEBUG "result": [ 08:38:51 DEBUG { 08:38:51 DEBUG "metric": { 08:38:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:38:51 DEBUG }, 08:38:51 DEBUG "value": [ 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG [...] 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:38:51 DEBUG }, 08:38:51 DEBUG "value": [ 08:38:51 DEBUG 1684568331, 08:38:51 DEBUG "0" 08:38:51 DEBUG ] 08:38:51 DEBUG } 08:38:51 DEBUG ] 08:38:51 DEBUG } 08:38:51 DEBUG } 08:38:51 INFO [http_cmd]: http status code OK 08:38:51 DEBUG --- status code --- 08:38:51 DEBUG http status code is 200 (expected 200) 08:38:51 DEBUG --- http response --- 08:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:38:51 DEBUG ----- output ----- 08:38:51 DEBUG { 08:38:51 DEBUG "status": "success", 08:38:51 DEBUG "data": { 08:38:51 DEBUG "resultType": "vector", 08:38:51 DEBUG "result": [ 08:38:51 DEBUG { 08:38:51 DEBUG "metric": { 08:38:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:38:51 DEBUG }, 08:38:51 DEBUG "value": [ 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG [...] 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:38:51 DEBUG }, 08:38:51 DEBUG "value": [ 08:38:51 DEBUG 1684568331, 08:38:51 DEBUG "0" 08:38:51 DEBUG ] 08:38:51 DEBUG } 08:38:51 DEBUG ] 08:38:51 DEBUG } 08:38:51 DEBUG } 08:38:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568331" 08:38:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568331" 08:38:51 INFO [http_cmd]: http status code OK 08:38:51 DEBUG --- status code --- 08:38:51 DEBUG http status code is 200 (expected 200) 08:38:51 DEBUG --- http response --- 08:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:38:51 DEBUG ----- output ----- 08:38:51 DEBUG { 08:38:51 DEBUG "status": "success", 08:38:51 DEBUG "data": { 08:38:51 DEBUG "resultType": "vector", 08:38:51 DEBUG "result": [ 08:38:51 DEBUG { 08:38:51 DEBUG "metric": { 08:38:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:38:51 DEBUG }, 08:38:51 DEBUG "value": [ 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG [...] 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:38:51 DEBUG }, 08:38:51 DEBUG "value": [ 08:38:51 DEBUG 1684568331, 08:38:51 DEBUG "1.031046355490201" 08:38:51 DEBUG ] 08:38:51 DEBUG } 08:38:51 DEBUG ] 08:38:51 DEBUG } 08:38:51 DEBUG } 08:38:51 INFO [http_cmd]: http status code OK 08:38:51 DEBUG --- status code --- 08:38:51 DEBUG http status code is 200 (expected 200) 08:38:51 DEBUG --- http response --- 08:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:38:51 DEBUG ----- output ----- 08:38:51 DEBUG { 08:38:51 DEBUG "status": "success", 08:38:51 DEBUG "data": { 08:38:51 DEBUG "resultType": "vector", 08:38:51 DEBUG "result": [ 08:38:51 DEBUG { 08:38:51 DEBUG "metric": { 08:38:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:38:51 DEBUG }, 08:38:51 DEBUG "value": [ 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG [...] 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG 08:38:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:38:51 DEBUG }, 08:38:51 DEBUG "value": [ 08:38:51 DEBUG 1684568331, 08:38:51 DEBUG "0" 08:38:51 DEBUG ] 08:38:51 DEBUG } 08:38:51 DEBUG ] 08:38:51 DEBUG } 08:38:51 DEBUG } 08:39:27 INFO 08:39:27 INFO [loop_until]: kubectl --namespace=xlou top pods 08:39:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:39:27 INFO [loop_until]: OK (rc = 0) 08:39:27 DEBUG --- stdout --- 08:39:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 363m 9404Mi am-7849cf7bdb-lxfkl 312m 9342Mi am-7849cf7bdb-td7js 321m 9402Mi ds-cts-0 632m 13619Mi ds-cts-1 705m 13831Mi ds-cts-2 854m 13763Mi ds-idrepo-0 236m 13661Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 999m 3209Mi 08:39:27 DEBUG --- stderr --- 08:39:27 DEBUG 08:39:39 INFO 08:39:39 INFO [loop_until]: kubectl --namespace=xlou top node 08:39:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:39:39 INFO [loop_until]: OK (rc = 0) 08:39:39 DEBUG --- stdout --- 08:39:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 381m 2% 11560Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 73m 0% 4943Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 472m 2% 11680Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 426m 2% 12006Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 151m 0% 3129Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 83m 0% 1362Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 86m 0% 5301Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 65m 0% 14644Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 759m 4% 15201Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 276m 1% 14771Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 66m 0% 14578Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 924m 5% 15110Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 919m 5% 14994Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1083m 6% 5255Mi 8% 08:39:39 DEBUG --- stderr --- 08:39:39 DEBUG 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "ds-idrepo-2" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "ds-idrepo-1" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG }, 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG "value": [ 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG "0" 08:39:50 DEBUG { 08:39:50 DEBUG ] 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG } 08:39:50 DEBUG "data": { 08:39:50 DEBUG ] 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG } 08:39:50 DEBUG "result": [ 08:39:50 DEBUG } 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "107902714.79383764" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "ds-idrepo-2" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "ds-idrepo-1" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "ds-cts-2" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "ds-idrepo-1" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "ds-cts-2" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "ds-idrepo-1" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "ds-idrepo-2" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "ds-idrepo-1" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "ds-cts-2" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "ds-idrepo-1" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568390" 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568390" 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568390" 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "instance": "10.142.0.17:9100" 08:39:50 DEBUG "instance": "10.142.0.17:9100" 08:39:50 DEBUG }, 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "instance": "10.142.0.61:9100" 08:39:50 DEBUG "instance": "10.142.0.61:9100" 08:39:50 DEBUG }, 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "6718.766666666666" 08:39:50 DEBUG "0.00036666666666747006" 08:39:50 DEBUG ] 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:39:50 DEBUG } 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:39:50 DEBUG }, 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG "value": [ 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:39:50 DEBUG [...] 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG 08:39:50 DEBUG { 08:39:50 DEBUG 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG 08:39:50 DEBUG "data": { 08:39:50 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG }, 08:39:50 DEBUG "result": [ 08:39:50 DEBUG "value": [ 08:39:50 DEBUG { 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "0" 08:39:50 DEBUG "instance": "10.142.0.17:9100" 08:39:50 DEBUG ] 08:39:50 DEBUG }, 08:39:50 DEBUG } 08:39:50 DEBUG "value": [ 08:39:50 DEBUG ] 08:39:50 DEBUG 08:39:50 DEBUG } 08:39:50 DEBUG 08:39:50 DEBUG } 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "instance": "10.142.0.61:9100" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568390" 08:39:50 INFO [http_cmd]: http status code OK 08:39:50 DEBUG --- status code --- 08:39:50 DEBUG http status code is 200 (expected 200) 08:39:50 DEBUG --- http response --- 08:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:39:50 DEBUG ----- output ----- 08:39:50 DEBUG { 08:39:50 DEBUG "status": "success", 08:39:50 DEBUG "data": { 08:39:50 DEBUG "resultType": "vector", 08:39:50 DEBUG "result": [ 08:39:50 DEBUG { 08:39:50 DEBUG "metric": { 08:39:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG [...] 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG 08:39:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:39:50 DEBUG }, 08:39:50 DEBUG "value": [ 08:39:50 DEBUG 1684568390, 08:39:50 DEBUG "0.2" 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG ] 08:39:50 DEBUG } 08:39:50 DEBUG } 08:39:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568391" 08:39:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568391" 08:39:51 INFO [http_cmd]: http status code OK 08:39:51 DEBUG --- status code --- 08:39:51 DEBUG http status code is 200 (expected 200) 08:39:51 DEBUG --- http response --- 08:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:39:51 DEBUG ----- output ----- 08:39:51 DEBUG { 08:39:51 DEBUG "status": "success", 08:39:51 DEBUG "data": { 08:39:51 DEBUG "resultType": "vector", 08:39:51 DEBUG "result": [ 08:39:51 DEBUG { 08:39:51 DEBUG "metric": { 08:39:51 DEBUG "instance": "10.142.0.17:9100" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG [...] 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG "instance": "10.142.0.61:9100" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 1684568391, 08:39:51 DEBUG "3159.7333333333336" 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG } 08:39:51 INFO [http_cmd]: http status code OK 08:39:51 DEBUG --- status code --- 08:39:51 DEBUG http status code is 200 (expected 200) 08:39:51 DEBUG --- http response --- 08:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:39:51 DEBUG ----- output ----- 08:39:51 DEBUG { 08:39:51 DEBUG "status": "success", 08:39:51 DEBUG "data": { 08:39:51 DEBUG "resultType": "vector", 08:39:51 DEBUG "result": [ 08:39:51 DEBUG { 08:39:51 DEBUG "metric": { 08:39:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG [...] 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 1684568391, 08:39:51 DEBUG "14521147392" 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG } 08:39:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568391" 08:39:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568391" 08:39:51 INFO [http_cmd]: http status code OK 08:39:51 DEBUG --- status code --- 08:39:51 DEBUG http status code is 200 (expected 200) 08:39:51 DEBUG --- http response --- 08:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:39:51 DEBUG ----- output ----- 08:39:51 DEBUG { 08:39:51 DEBUG "status": "success", 08:39:51 DEBUG "data": { 08:39:51 DEBUG "resultType": "vector", 08:39:51 DEBUG "result": [ 08:39:51 DEBUG { 08:39:51 DEBUG "metric": { 08:39:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG [...] 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 1684568391, 08:39:51 DEBUG "0" 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 INFO [http_cmd]: http status code OK 08:39:51 DEBUG } 08:39:51 DEBUG --- status code --- 08:39:51 DEBUG http status code is 200 (expected 200) 08:39:51 DEBUG --- http response --- 08:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:39:51 DEBUG ----- output ----- 08:39:51 DEBUG { 08:39:51 DEBUG "status": "success", 08:39:51 DEBUG "data": { 08:39:51 DEBUG "resultType": "vector", 08:39:51 DEBUG "result": [ 08:39:51 DEBUG { 08:39:51 DEBUG "metric": { 08:39:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG [...] 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 1684568391, 08:39:51 DEBUG "0" 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG } 08:39:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568391" 08:39:51 INFO [http_cmd]: http status code OK 08:39:51 DEBUG --- status code --- 08:39:51 DEBUG http status code is 200 (expected 200) 08:39:51 DEBUG --- http response --- 08:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:39:51 DEBUG ----- output ----- 08:39:51 DEBUG { 08:39:51 DEBUG "status": "success", 08:39:51 DEBUG "data": { 08:39:51 DEBUG "resultType": "vector", 08:39:51 DEBUG "result": [ 08:39:51 DEBUG { 08:39:51 DEBUG "metric": { 08:39:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG [...] 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 1684568391, 08:39:51 DEBUG "0.0028921274814895114" 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG } 08:39:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568391" 08:39:51 INFO [http_cmd]: http status code OK 08:39:51 DEBUG --- status code --- 08:39:51 DEBUG http status code is 200 (expected 200) 08:39:51 DEBUG --- http response --- 08:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:39:51 DEBUG ----- output ----- 08:39:51 DEBUG { 08:39:51 DEBUG "status": "success", 08:39:51 DEBUG "data": { 08:39:51 DEBUG "resultType": "vector", 08:39:51 DEBUG "result": [ 08:39:51 DEBUG { 08:39:51 DEBUG "metric": { 08:39:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG [...] 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG 08:39:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:39:51 DEBUG }, 08:39:51 DEBUG "value": [ 08:39:51 DEBUG 1684568391, 08:39:51 DEBUG "1.0294854485687792" 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG ] 08:39:51 DEBUG } 08:39:51 DEBUG } 08:40:27 INFO 08:40:27 INFO [loop_until]: kubectl --namespace=xlou top pods 08:40:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:40:27 INFO [loop_until]: OK (rc = 0) 08:40:27 DEBUG --- stdout --- 08:40:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 397m 9403Mi am-7849cf7bdb-lxfkl 305m 9342Mi am-7849cf7bdb-td7js 310m 9402Mi ds-cts-0 584m 13623Mi ds-cts-1 715m 13851Mi ds-cts-2 823m 13784Mi ds-idrepo-0 237m 13662Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1016m 3424Mi 08:40:27 DEBUG --- stderr --- 08:40:27 DEBUG 08:40:39 INFO 08:40:39 INFO [loop_until]: kubectl --namespace=xlou top node 08:40:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:40:39 INFO [loop_until]: OK (rc = 0) 08:40:39 DEBUG --- stdout --- 08:40:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 389m 2% 11560Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 73m 0% 4947Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 502m 3% 11677Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 407m 2% 12008Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 153m 0% 3140Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 88m 0% 1358Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 85m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 62m 0% 14647Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 753m 4% 15217Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 280m 1% 14769Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14577Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 957m 6% 15132Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 652m 4% 14994Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1079m 6% 5492Mi 9% 08:40:39 DEBUG --- stderr --- 08:40:39 DEBUG 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "pod": "ds-idrepo-2" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "pod": "ds-idrepo-1" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "0" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG { 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG "data": { 08:40:50 DEBUG { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG "data": { 08:40:50 DEBUG { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "result": [ 08:40:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:40:50 DEBUG { 08:40:50 DEBUG }, 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "value": [ 08:40:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:40:50 DEBUG 08:40:50 DEBUG }, 08:40:50 DEBUG 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:40:50 DEBUG 08:40:50 DEBUG }, 08:40:50 DEBUG 08:40:50 DEBUG "value": [ 08:40:50 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG }, 08:40:50 DEBUG "0" 08:40:50 DEBUG "value": [ 08:40:50 DEBUG ] 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG } 08:40:50 DEBUG "0" 08:40:50 DEBUG ] 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "pod": "ds-idrepo-2" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "pod": "ds-idrepo-1" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "0" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "pod": "ds-cts-2" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "pod": "ds-idrepo-1" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "0" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "pod": "ds-cts-2" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "pod": "ds-idrepo-1" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "0" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "0" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "pod": "ds-idrepo-2" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "pod": "ds-idrepo-1" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "0" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568450" 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "instance": "10.142.0.17:9100" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "instance": "10.142.0.61:9100" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "6625.866666666667" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568450" 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "pod": "ds-cts-2" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "pod": "ds-idrepo-1" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "0" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "instance": "10.142.0.17:9100" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "instance": "10.142.0.61:9100" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "0.0001666666666665151" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568450" 08:40:50 INFO [http_cmd]: http status code OK 08:40:50 DEBUG --- status code --- 08:40:50 DEBUG http status code is 200 (expected 200) 08:40:50 DEBUG --- http response --- 08:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:40:50 DEBUG ----- output ----- 08:40:50 DEBUG { 08:40:50 DEBUG "status": "success", 08:40:50 DEBUG "data": { 08:40:50 DEBUG "resultType": "vector", 08:40:50 DEBUG "result": [ 08:40:50 DEBUG { 08:40:50 DEBUG "metric": { 08:40:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG [...] 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG 08:40:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:40:50 DEBUG }, 08:40:50 DEBUG "value": [ 08:40:50 DEBUG 1684568450, 08:40:50 DEBUG "0.2" 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG ] 08:40:50 DEBUG } 08:40:50 DEBUG } 08:40:51 INFO [http_cmd]: http status code OK 08:40:51 DEBUG --- status code --- 08:40:51 DEBUG http status code is 200 (expected 200) 08:40:51 DEBUG --- http response --- 08:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:40:51 DEBUG ----- output ----- 08:40:51 DEBUG { 08:40:51 DEBUG "status": "success", 08:40:51 DEBUG "data": { 08:40:51 DEBUG "resultType": "vector", 08:40:51 DEBUG "result": [ 08:40:51 DEBUG { 08:40:51 DEBUG "metric": { 08:40:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG [...] 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 1684568450, 08:40:51 DEBUG "0" 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568451" 08:40:51 INFO [http_cmd]: http status code OK 08:40:51 DEBUG --- status code --- 08:40:51 DEBUG http status code is 200 (expected 200) 08:40:51 DEBUG --- http response --- 08:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:40:51 DEBUG ----- output ----- 08:40:51 DEBUG { 08:40:51 DEBUG "status": "success", 08:40:51 DEBUG "data": { 08:40:51 DEBUG "resultType": "vector", 08:40:51 DEBUG "result": [ 08:40:51 DEBUG { 08:40:51 DEBUG "metric": { 08:40:51 DEBUG "instance": "10.142.0.17:9100" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG [...] 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG "instance": "10.142.0.61:9100" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 1684568451, 08:40:51 DEBUG "0.000333333333333267" 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568451" 08:40:51 INFO [http_cmd]: http status code OK 08:40:51 DEBUG --- status code --- 08:40:51 DEBUG http status code is 200 (expected 200) 08:40:51 DEBUG --- http response --- 08:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:40:51 DEBUG ----- output ----- 08:40:51 DEBUG { 08:40:51 DEBUG "status": "success", 08:40:51 DEBUG "data": { 08:40:51 DEBUG "resultType": "vector", 08:40:51 DEBUG "result": [ 08:40:51 DEBUG { 08:40:51 DEBUG "metric": { 08:40:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG [...] 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 1684568451, 08:40:51 DEBUG "0" 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568451" 08:40:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568451" 08:40:51 INFO [http_cmd]: http status code OK 08:40:51 DEBUG --- status code --- 08:40:51 DEBUG http status code is 200 (expected 200) 08:40:51 DEBUG --- http response --- 08:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:40:51 DEBUG ----- output ----- 08:40:51 DEBUG { 08:40:51 DEBUG "status": "success", 08:40:51 DEBUG "data": { 08:40:51 DEBUG "resultType": "vector", 08:40:51 DEBUG "result": [ 08:40:51 DEBUG { 08:40:51 DEBUG "metric": { 08:40:51 DEBUG "instance": "10.142.0.17:9100" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG [...] 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG "instance": "10.142.0.61:9100" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 1684568451, 08:40:51 DEBUG "3009.9666666666667" 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 INFO [http_cmd]: http status code OK 08:40:51 DEBUG --- status code --- 08:40:51 DEBUG http status code is 200 (expected 200) 08:40:51 DEBUG --- http response --- 08:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:40:51 DEBUG ----- output ----- 08:40:51 DEBUG { 08:40:51 DEBUG "status": "success", 08:40:51 DEBUG "data": { 08:40:51 DEBUG "resultType": "vector", 08:40:51 DEBUG "result": [ 08:40:51 DEBUG { 08:40:51 DEBUG "metric": { 08:40:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG [...] 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 1684568451, 08:40:51 DEBUG "14457700352" 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568451" 08:40:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568451" 08:40:51 INFO [http_cmd]: http status code OK 08:40:51 INFO [http_cmd]: http status code OK 08:40:51 DEBUG --- status code --- 08:40:51 DEBUG --- status code --- 08:40:51 DEBUG http status code is 200 (expected 200) 08:40:51 DEBUG http status code is 200 (expected 200) 08:40:51 DEBUG --- http response --- 08:40:51 DEBUG --- http response --- 08:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:40:51 DEBUG ----- output ----- 08:40:51 DEBUG ----- output ----- 08:40:51 DEBUG { 08:40:51 DEBUG { 08:40:51 DEBUG "status": "success", 08:40:51 DEBUG "status": "success", 08:40:51 DEBUG "data": { 08:40:51 DEBUG "data": { 08:40:51 DEBUG "resultType": "vector", 08:40:51 DEBUG "resultType": "vector", 08:40:51 DEBUG "result": [ 08:40:51 DEBUG "result": [ 08:40:51 DEBUG { 08:40:51 DEBUG { 08:40:51 DEBUG "metric": { 08:40:51 DEBUG "metric": { 08:40:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:40:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:40:51 DEBUG }, 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG [...] 08:40:51 DEBUG [...] 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:40:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:40:51 DEBUG }, 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 1684568451, 08:40:51 DEBUG 1684568451, 08:40:51 DEBUG "0" 08:40:51 DEBUG "0" 08:40:51 DEBUG ] 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 DEBUG ] 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568451" 08:40:51 INFO [http_cmd]: http status code OK 08:40:51 DEBUG --- status code --- 08:40:51 DEBUG http status code is 200 (expected 200) 08:40:51 DEBUG --- http response --- 08:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:40:51 DEBUG ----- output ----- 08:40:51 DEBUG { 08:40:51 DEBUG "status": "success", 08:40:51 DEBUG "data": { 08:40:51 DEBUG "resultType": "vector", 08:40:51 DEBUG "result": [ 08:40:51 DEBUG { 08:40:51 DEBUG "metric": { 08:40:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG [...] 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 1684568451, 08:40:51 DEBUG "0" 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG } 08:40:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568451" 08:40:51 INFO [http_cmd]: http status code OK 08:40:51 DEBUG --- status code --- 08:40:51 DEBUG http status code is 200 (expected 200) 08:40:51 DEBUG --- http response --- 08:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:40:51 DEBUG ----- output ----- 08:40:51 DEBUG { 08:40:51 DEBUG "status": "success", 08:40:51 DEBUG "data": { 08:40:51 DEBUG "resultType": "vector", 08:40:51 DEBUG "result": [ 08:40:51 DEBUG { 08:40:51 DEBUG "metric": { 08:40:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG [...] 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG 08:40:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:40:51 DEBUG }, 08:40:51 DEBUG "value": [ 08:40:51 DEBUG 1684568451, 08:40:51 DEBUG "1.024912160426797" 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG ] 08:40:51 DEBUG } 08:40:51 DEBUG } 08:41:27 INFO 08:41:27 INFO [loop_until]: kubectl --namespace=xlou top pods 08:41:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:41:27 INFO [loop_until]: OK (rc = 0) 08:41:27 DEBUG --- stdout --- 08:41:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 369m 9403Mi am-7849cf7bdb-lxfkl 313m 9342Mi am-7849cf7bdb-td7js 300m 9402Mi ds-cts-0 627m 13501Mi ds-cts-1 998m 13787Mi ds-cts-2 648m 13704Mi ds-idrepo-0 233m 13662Mi ds-idrepo-1 13m 13554Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 6m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1003m 3702Mi 08:41:27 DEBUG --- stderr --- 08:41:27 DEBUG 08:41:39 INFO 08:41:39 INFO [loop_until]: kubectl --namespace=xlou top node 08:41:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:41:39 INFO [loop_until]: OK (rc = 0) 08:41:39 DEBUG --- stdout --- 08:41:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 399m 2% 11561Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 72m 0% 4947Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 486m 3% 11676Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 404m 2% 12007Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 156m 0% 3136Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 81m 0% 1364Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 87m 0% 5301Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 63m 0% 14645Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1000m 6% 15165Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 282m 1% 14769Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14574Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 690m 4% 15054Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 778m 4% 14866Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1079m 6% 5729Mi 9% 08:41:39 DEBUG --- stderr --- 08:41:39 DEBUG 08:41:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568510" 08:41:50 INFO [http_cmd]: http status code OK 08:41:50 DEBUG --- status code --- 08:41:50 DEBUG http status code is 200 (expected 200) 08:41:50 DEBUG --- http response --- 08:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:41:50 DEBUG ----- output ----- 08:41:50 DEBUG { 08:41:50 DEBUG "status": "success", 08:41:50 DEBUG "data": { 08:41:50 DEBUG "resultType": "vector", 08:41:50 DEBUG "result": [ 08:41:50 DEBUG { 08:41:50 DEBUG "metric": { 08:41:50 DEBUG "pod": "ds-idrepo-2" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG [...] 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG "pod": "ds-idrepo-1" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 1684568510, 08:41:50 DEBUG "0" 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG } 08:41:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568510" 08:41:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568510" 08:41:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568510" 08:41:50 INFO [http_cmd]: http status code OK 08:41:50 DEBUG --- status code --- 08:41:50 DEBUG http status code is 200 (expected 200) 08:41:50 DEBUG --- http response --- 08:41:50 INFO [http_cmd]: http status code OK 08:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:41:50 DEBUG --- status code --- 08:41:50 DEBUG ----- output ----- 08:41:50 DEBUG http status code is 200 (expected 200) 08:41:50 DEBUG { 08:41:50 DEBUG --- http response --- 08:41:50 DEBUG "status": "success", 08:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:41:50 DEBUG "data": { 08:41:50 DEBUG ----- output ----- 08:41:50 DEBUG "resultType": "vector", 08:41:50 DEBUG { 08:41:50 DEBUG "result": [ 08:41:50 DEBUG "status": "success", 08:41:50 DEBUG { 08:41:50 DEBUG "data": { 08:41:50 DEBUG "metric": { 08:41:50 DEBUG "resultType": "vector", 08:41:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:41:50 DEBUG "result": [ 08:41:50 DEBUG }, 08:41:50 DEBUG { 08:41:50 DEBUG "value": [ 08:41:50 DEBUG "metric": { 08:41:50 DEBUG 08:41:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:41:50 DEBUG 08:41:50 DEBUG }, 08:41:50 DEBUG 08:41:50 DEBUG "value": [ 08:41:50 DEBUG [...] 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG [...] 08:41:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:41:50 DEBUG 08:41:50 DEBUG }, 08:41:50 DEBUG 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 08:41:50 DEBUG 1684568510, 08:41:50 DEBUG "pod": "ds-idrepo-1" 08:41:50 DEBUG "0" 08:41:50 DEBUG }, 08:41:50 DEBUG ] 08:41:50 DEBUG "value": [ 08:41:50 DEBUG } 08:41:50 DEBUG 1684568510, 08:41:50 DEBUG ] 08:41:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568510" 08:41:50 DEBUG "0" 08:41:50 INFO [http_cmd]: http status code OK 08:41:50 DEBUG } 08:41:50 DEBUG ] 08:41:50 DEBUG --- status code --- 08:41:50 DEBUG } 08:41:50 DEBUG } 08:41:50 DEBUG http status code is 200 (expected 200) 08:41:50 DEBUG ] 08:41:50 DEBUG --- http response --- 08:41:50 DEBUG } 08:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:41:50 DEBUG } 08:41:50 DEBUG ----- output ----- 08:41:50 DEBUG { 08:41:50 DEBUG "status": "success", 08:41:50 DEBUG "data": { 08:41:50 DEBUG "resultType": "vector", 08:41:50 DEBUG "result": [ 08:41:50 DEBUG { 08:41:50 DEBUG "metric": { 08:41:50 DEBUG "pod": "ds-cts-2" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG [...] 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG "pod": "ds-idrepo-1" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 1684568510, 08:41:50 DEBUG "0" 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG } 08:41:50 INFO [http_cmd]: http status code OK 08:41:50 DEBUG --- status code --- 08:41:50 DEBUG http status code is 200 (expected 200) 08:41:50 DEBUG --- http response --- 08:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:41:50 DEBUG ----- output ----- 08:41:50 DEBUG { 08:41:50 DEBUG "status": "success", 08:41:50 DEBUG "data": { 08:41:50 DEBUG "resultType": "vector", 08:41:50 DEBUG "result": [ 08:41:50 DEBUG { 08:41:50 DEBUG "metric": { 08:41:50 DEBUG "pod": "ds-idrepo-2" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG [...] 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG "pod": "ds-idrepo-1" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 1684568510, 08:41:50 DEBUG "0" 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG } 08:41:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568510" 08:41:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568510" 08:41:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568510" 08:41:50 INFO [http_cmd]: http status code OK 08:41:50 DEBUG --- status code --- 08:41:50 DEBUG http status code is 200 (expected 200) 08:41:50 DEBUG --- http response --- 08:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:41:50 DEBUG ----- output ----- 08:41:50 DEBUG { 08:41:50 DEBUG "status": "success", 08:41:50 DEBUG "data": { 08:41:50 DEBUG "resultType": "vector", 08:41:50 DEBUG "result": [ 08:41:50 DEBUG { 08:41:50 DEBUG "metric": { 08:41:50 DEBUG "pod": "ds-cts-2" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG [...] 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG "pod": "ds-idrepo-1" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 1684568510, 08:41:50 DEBUG "0" 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG } 08:41:50 INFO [http_cmd]: http status code OK 08:41:50 DEBUG --- status code --- 08:41:50 DEBUG http status code is 200 (expected 200) 08:41:50 DEBUG --- http response --- 08:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:41:50 DEBUG ----- output ----- 08:41:50 DEBUG { 08:41:50 DEBUG "status": "success", 08:41:50 DEBUG "data": { 08:41:50 DEBUG "resultType": "vector", 08:41:50 DEBUG "result": [ 08:41:50 DEBUG { 08:41:50 DEBUG "metric": { 08:41:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG [...] 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 1684568510, 08:41:50 DEBUG "0" 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG } 08:41:50 INFO [http_cmd]: http status code OK 08:41:50 DEBUG --- status code --- 08:41:50 DEBUG http status code is 200 (expected 200) 08:41:50 DEBUG --- http response --- 08:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:41:50 DEBUG ----- output ----- 08:41:50 DEBUG { 08:41:50 DEBUG "status": "success", 08:41:50 DEBUG "data": { 08:41:50 DEBUG "resultType": "vector", 08:41:50 DEBUG "result": [ 08:41:50 DEBUG { 08:41:50 DEBUG "metric": { 08:41:50 DEBUG "pod": "ds-idrepo-2" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG [...] 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG 08:41:50 DEBUG "pod": "ds-idrepo-1" 08:41:50 DEBUG }, 08:41:50 DEBUG "value": [ 08:41:50 DEBUG 1684568510, 08:41:50 DEBUG "0" 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG ] 08:41:50 DEBUG } 08:41:50 DEBUG } 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568511" 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "pod": "ds-cts-2" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "pod": "ds-idrepo-1" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "0" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568511" 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "instance": "10.142.0.17:9100" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "instance": "10.142.0.61:9100" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "6654.266666666667" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568511" 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568511" 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "0.2" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "instance": "10.142.0.17:9100" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "instance": "10.142.0.61:9100" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "0.0004999999999995453" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568511" 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "pod": "ds-cts-0" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "162.98472811695143" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568511" 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568511" 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "instance": "10.142.0.17:9100" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "instance": "10.142.0.61:9100" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "0" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "0" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568511" 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "14479507456" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568511" 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "instance": "10.142.0.17:9100" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "instance": "10.142.0.61:9100" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "2911.0333333333333" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568511" 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568511" 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568511" 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:41:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:41:51 DEBUG }, 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "pod": "ds-idrepo-1" 08:41:51 DEBUG "pod": "ds-cts-0" 08:41:51 DEBUG }, 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "0" 08:41:51 DEBUG "4.534407080930097" 08:41:51 DEBUG ] 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "0" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:41:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568511" 08:41:51 INFO [http_cmd]: http status code OK 08:41:51 DEBUG --- status code --- 08:41:51 DEBUG http status code is 200 (expected 200) 08:41:51 DEBUG --- http response --- 08:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:41:51 DEBUG ----- output ----- 08:41:51 DEBUG { 08:41:51 DEBUG "status": "success", 08:41:51 DEBUG "data": { 08:41:51 DEBUG "resultType": "vector", 08:41:51 DEBUG "result": [ 08:41:51 DEBUG { 08:41:51 DEBUG "metric": { 08:41:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG [...] 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG 08:41:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:41:51 DEBUG }, 08:41:51 DEBUG "value": [ 08:41:51 DEBUG 1684568511, 08:41:51 DEBUG "1.0314947366084795" 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG ] 08:41:51 DEBUG } 08:41:51 DEBUG } 08:42:27 INFO 08:42:27 INFO [loop_until]: kubectl --namespace=xlou top pods 08:42:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:42:27 INFO [loop_until]: OK (rc = 0) 08:42:27 DEBUG --- stdout --- 08:42:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 358m 9403Mi am-7849cf7bdb-lxfkl 304m 9342Mi am-7849cf7bdb-td7js 329m 9403Mi ds-cts-0 854m 13533Mi ds-cts-1 1068m 13835Mi ds-cts-2 899m 13734Mi ds-idrepo-0 236m 13662Mi ds-idrepo-1 13m 13555Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 8m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 992m 3949Mi 08:42:27 DEBUG --- stderr --- 08:42:27 DEBUG 08:42:39 INFO 08:42:39 INFO [loop_until]: kubectl --namespace=xlou top node 08:42:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:42:39 INFO [loop_until]: OK (rc = 0) 08:42:39 DEBUG --- stdout --- 08:42:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 403m 2% 11572Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 4951Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 486m 3% 11673Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 406m 2% 12009Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 175m 1% 3102Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 87m 0% 1364Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 94m 0% 5301Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14645Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 983m 6% 15231Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 278m 1% 14768Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14578Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 1282m 8% 15020Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 819m 5% 14880Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1075m 6% 6083Mi 10% 08:42:39 DEBUG --- stderr --- 08:42:39 DEBUG 08:42:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568570" 08:42:50 INFO [http_cmd]: http status code OK 08:42:50 DEBUG --- status code --- 08:42:50 DEBUG http status code is 200 (expected 200) 08:42:50 DEBUG --- http response --- 08:42:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:42:50 DEBUG ----- output ----- 08:42:50 DEBUG { 08:42:50 DEBUG "status": "success", 08:42:50 DEBUG "data": { 08:42:50 DEBUG "resultType": "vector", 08:42:50 DEBUG "result": [ 08:42:50 DEBUG { 08:42:50 DEBUG "metric": { 08:42:50 DEBUG "pod": "ds-idrepo-2" 08:42:50 DEBUG }, 08:42:50 DEBUG "value": [ 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG [...] 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG "pod": "ds-idrepo-1" 08:42:50 DEBUG }, 08:42:50 DEBUG "value": [ 08:42:50 DEBUG 1684568570, 08:42:50 DEBUG "0" 08:42:50 DEBUG ] 08:42:50 DEBUG } 08:42:50 DEBUG ] 08:42:50 DEBUG } 08:42:50 DEBUG } 08:42:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568570" 08:42:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568570" 08:42:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568570" 08:42:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568570" 08:42:50 INFO [http_cmd]: http status code OK 08:42:50 DEBUG --- status code --- 08:42:50 INFO [http_cmd]: http status code OK 08:42:50 DEBUG http status code is 200 (expected 200) 08:42:50 DEBUG --- status code --- 08:42:50 DEBUG http status code is 200 (expected 200) 08:42:50 DEBUG --- http response --- 08:42:50 DEBUG --- http response --- 08:42:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:42:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:42:50 DEBUG ----- output ----- 08:42:50 DEBUG ----- output ----- 08:42:50 INFO [http_cmd]: http status code OK 08:42:50 DEBUG { 08:42:50 DEBUG { 08:42:50 DEBUG --- status code --- 08:42:50 DEBUG "status": "success", 08:42:50 DEBUG "status": "success", 08:42:50 DEBUG http status code is 200 (expected 200) 08:42:50 DEBUG "data": { 08:42:50 DEBUG "data": { 08:42:50 DEBUG --- http response --- 08:42:50 DEBUG "resultType": "vector", 08:42:50 DEBUG "resultType": "vector", 08:42:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:42:50 DEBUG "result": [ 08:42:50 DEBUG "result": [ 08:42:50 DEBUG ----- output ----- 08:42:50 DEBUG { 08:42:50 DEBUG { 08:42:50 DEBUG { 08:42:50 DEBUG "metric": { 08:42:50 DEBUG "metric": { 08:42:50 DEBUG "status": "success", 08:42:50 DEBUG "pod": "ds-cts-2" 08:42:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:42:50 DEBUG "data": { 08:42:50 DEBUG }, 08:42:50 DEBUG }, 08:42:50 DEBUG "resultType": "vector", 08:42:50 DEBUG "value": [ 08:42:50 DEBUG "value": [ 08:42:50 DEBUG "result": [ 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG { 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG "metric": { 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:42:50 DEBUG [...] 08:42:50 DEBUG [...] 08:42:50 DEBUG }, 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG "value": [ 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG "pod": "ds-idrepo-1" 08:42:50 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:42:50 DEBUG 08:42:50 DEBUG }, 08:42:50 DEBUG }, 08:42:50 DEBUG [...] 08:42:50 DEBUG "value": [ 08:42:50 DEBUG "value": [ 08:42:50 DEBUG 08:42:50 DEBUG 1684568570, 08:42:50 DEBUG 1684568570, 08:42:50 DEBUG 08:42:50 DEBUG "0" 08:42:50 DEBUG "326.08971685751726" 08:42:50 DEBUG 08:42:50 DEBUG ] 08:42:50 DEBUG ] 08:42:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:42:50 DEBUG } 08:42:50 DEBUG } 08:42:50 DEBUG }, 08:42:50 DEBUG ] 08:42:50 DEBUG ] 08:42:50 DEBUG "value": [ 08:42:50 INFO [http_cmd]: http status code OK 08:42:50 DEBUG } 08:42:50 DEBUG } 08:42:50 DEBUG 1684568570, 08:42:50 DEBUG --- status code --- 08:42:50 DEBUG } 08:42:50 DEBUG } 08:42:50 DEBUG "0" 08:42:50 DEBUG http status code is 200 (expected 200) 08:42:50 DEBUG ] 08:42:50 DEBUG --- http response --- 08:42:50 DEBUG } 08:42:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:42:50 DEBUG ] 08:42:50 DEBUG ----- output ----- 08:42:50 DEBUG } 08:42:50 DEBUG { 08:42:50 DEBUG } 08:42:50 DEBUG "status": "success", 08:42:50 DEBUG "data": { 08:42:50 DEBUG "resultType": "vector", 08:42:50 DEBUG "result": [ 08:42:50 DEBUG { 08:42:50 DEBUG "metric": { 08:42:50 DEBUG "pod": "ds-idrepo-2" 08:42:50 DEBUG }, 08:42:50 DEBUG "value": [ 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG [...] 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG 08:42:50 DEBUG "pod": "ds-idrepo-1" 08:42:50 DEBUG }, 08:42:50 DEBUG "value": [ 08:42:50 DEBUG 1684568570, 08:42:50 DEBUG "0" 08:42:50 DEBUG ] 08:42:50 DEBUG } 08:42:50 DEBUG ] 08:42:50 DEBUG } 08:42:50 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "pod": "ds-idrepo-2" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "pod": "ds-idrepo-1" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "0" 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "pod": "ds-cts-2" 08:42:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:42:51 DEBUG }, 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "pod": "ds-idrepo-1" 08:42:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:42:51 DEBUG }, 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG "value": [ 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568571" 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "0" 08:42:51 DEBUG "0" 08:42:51 DEBUG ] 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "instance": "10.142.0.17:9100" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "instance": "10.142.0.61:9100" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "0.0004333333333325603" 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "pod": "ds-cts-2" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "pod": "ds-idrepo-1" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "0" 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "instance": "10.142.0.17:9100" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "instance": "10.142.0.61:9100" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "6787.6" 08:42:51 DEBUG ] 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:42:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:42:51 DEBUG }, 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:42:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:42:51 DEBUG }, 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "0" 08:42:51 DEBUG "0" 08:42:51 DEBUG ] 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "0.2" 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "instance": "10.142.0.17:9100" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "instance": "10.142.0.61:9100" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "0" 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "14429560832" 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "instance": "10.142.0.17:9100" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "instance": "10.142.0.61:9100" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "5075.566666666667" 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "0.4510767461106621" 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG { 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG "data": { 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "data": { 08:42:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG }, 08:42:51 DEBUG "result": [ 08:42:51 DEBUG "value": [ 08:42:51 DEBUG { 08:42:51 DEBUG 08:42:51 DEBUG "metric": { 08:42:51 DEBUG 08:42:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:42:51 DEBUG 08:42:51 DEBUG }, 08:42:51 DEBUG [...] 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:42:51 DEBUG [...] 08:42:51 DEBUG }, 08:42:51 DEBUG 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG 08:42:51 DEBUG "0" 08:42:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:42:51 DEBUG ] 08:42:51 DEBUG }, 08:42:51 DEBUG } 08:42:51 DEBUG "value": [ 08:42:51 DEBUG ] 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG } 08:42:51 DEBUG "0" 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:42:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568571" 08:42:51 INFO [http_cmd]: http status code OK 08:42:51 DEBUG --- status code --- 08:42:51 DEBUG http status code is 200 (expected 200) 08:42:51 DEBUG --- http response --- 08:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:42:51 DEBUG ----- output ----- 08:42:51 DEBUG { 08:42:51 DEBUG "status": "success", 08:42:51 DEBUG "data": { 08:42:51 DEBUG "resultType": "vector", 08:42:51 DEBUG "result": [ 08:42:51 DEBUG { 08:42:51 DEBUG "metric": { 08:42:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG [...] 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG 08:42:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:42:51 DEBUG }, 08:42:51 DEBUG "value": [ 08:42:51 DEBUG 1684568571, 08:42:51 DEBUG "1.0369031492958523" 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG ] 08:42:51 DEBUG } 08:42:51 DEBUG } 08:43:27 INFO 08:43:27 INFO [loop_until]: kubectl --namespace=xlou top pods 08:43:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:43:27 INFO [loop_until]: OK (rc = 0) 08:43:27 DEBUG --- stdout --- 08:43:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 362m 9403Mi am-7849cf7bdb-lxfkl 299m 9342Mi am-7849cf7bdb-td7js 309m 9403Mi ds-cts-0 992m 13534Mi ds-cts-1 671m 13875Mi ds-cts-2 637m 13674Mi ds-idrepo-0 225m 13662Mi ds-idrepo-1 13m 13555Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1015m 4136Mi 08:43:27 DEBUG --- stderr --- 08:43:27 DEBUG 08:43:39 INFO 08:43:39 INFO [loop_until]: kubectl --namespace=xlou top node 08:43:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:43:39 INFO [loop_until]: OK (rc = 0) 08:43:39 DEBUG --- stdout --- 08:43:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 383m 2% 11558Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4950Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 714m 4% 11674Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 429m 2% 12006Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 140m 0% 3130Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 86m 0% 1361Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 93m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 67m 0% 14644Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1239m 7% 14969Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 286m 1% 14772Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14576Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 939m 5% 15048Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 640m 4% 14892Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1081m 6% 6318Mi 10% 08:43:39 DEBUG --- stderr --- 08:43:39 DEBUG 08:43:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568630" 08:43:50 INFO [http_cmd]: http status code OK 08:43:50 DEBUG --- status code --- 08:43:50 DEBUG http status code is 200 (expected 200) 08:43:50 DEBUG --- http response --- 08:43:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:43:50 DEBUG ----- output ----- 08:43:50 DEBUG { 08:43:50 DEBUG "status": "success", 08:43:50 DEBUG "data": { 08:43:50 DEBUG "resultType": "vector", 08:43:50 DEBUG "result": [ 08:43:50 DEBUG { 08:43:50 DEBUG "metric": { 08:43:50 DEBUG "pod": "ds-idrepo-2" 08:43:50 DEBUG }, 08:43:50 DEBUG "value": [ 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG [...] 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG "pod": "ds-idrepo-1" 08:43:50 DEBUG }, 08:43:50 DEBUG "value": [ 08:43:50 DEBUG 1684568630, 08:43:50 DEBUG "0" 08:43:50 DEBUG ] 08:43:50 DEBUG } 08:43:50 DEBUG ] 08:43:50 DEBUG } 08:43:50 DEBUG } 08:43:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568630" 08:43:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568630" 08:43:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568630" 08:43:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568630" 08:43:50 INFO [http_cmd]: http status code OK 08:43:50 DEBUG --- status code --- 08:43:50 DEBUG http status code is 200 (expected 200) 08:43:50 DEBUG --- http response --- 08:43:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:43:50 DEBUG ----- output ----- 08:43:50 DEBUG { 08:43:50 DEBUG "status": "success", 08:43:50 DEBUG "data": { 08:43:50 DEBUG "resultType": "vector", 08:43:50 DEBUG "result": [ 08:43:50 DEBUG { 08:43:50 DEBUG "metric": { 08:43:50 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:43:50 DEBUG }, 08:43:50 DEBUG "value": [ 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG [...] 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG "pod": "ds-idrepo-1" 08:43:50 DEBUG }, 08:43:50 DEBUG "value": [ 08:43:50 DEBUG 1684568630, 08:43:50 DEBUG "0" 08:43:50 DEBUG ] 08:43:50 DEBUG } 08:43:50 DEBUG ] 08:43:50 DEBUG } 08:43:50 DEBUG } 08:43:50 INFO [http_cmd]: http status code OK 08:43:50 DEBUG --- status code --- 08:43:50 INFO [http_cmd]: http status code OK 08:43:50 INFO [http_cmd]: http status code OK 08:43:50 DEBUG http status code is 200 (expected 200) 08:43:50 DEBUG --- status code --- 08:43:50 DEBUG --- status code --- 08:43:50 DEBUG --- http response --- 08:43:50 DEBUG http status code is 200 (expected 200) 08:43:50 DEBUG http status code is 200 (expected 200) 08:43:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:43:50 DEBUG --- http response --- 08:43:50 DEBUG --- http response --- 08:43:50 DEBUG ----- output ----- 08:43:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:43:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:43:50 DEBUG { 08:43:50 DEBUG ----- output ----- 08:43:50 DEBUG ----- output ----- 08:43:50 DEBUG "status": "success", 08:43:50 DEBUG { 08:43:50 DEBUG { 08:43:50 DEBUG "data": { 08:43:50 DEBUG "status": "success", 08:43:50 DEBUG "status": "success", 08:43:50 DEBUG "resultType": "vector", 08:43:50 DEBUG "data": { 08:43:50 DEBUG "data": { 08:43:50 DEBUG "result": [ 08:43:50 DEBUG "resultType": "vector", 08:43:50 DEBUG "resultType": "vector", 08:43:50 DEBUG { 08:43:50 DEBUG "result": [ 08:43:50 DEBUG "result": [ 08:43:50 DEBUG "metric": { 08:43:50 DEBUG { 08:43:50 DEBUG { 08:43:50 DEBUG "pod": "ds-cts-2" 08:43:50 DEBUG "metric": { 08:43:50 DEBUG "metric": { 08:43:50 DEBUG }, 08:43:50 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:43:50 DEBUG "pod": "ds-idrepo-2" 08:43:50 DEBUG "value": [ 08:43:50 DEBUG }, 08:43:50 DEBUG }, 08:43:50 DEBUG 08:43:50 DEBUG "value": [ 08:43:50 DEBUG "value": [ 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG [...] 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG [...] 08:43:50 DEBUG [...] 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG "pod": "ds-idrepo-1" 08:43:50 DEBUG 08:43:50 DEBUG 08:43:50 DEBUG }, 08:43:50 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:43:50 DEBUG "pod": "ds-idrepo-1" 08:43:50 DEBUG "value": [ 08:43:50 DEBUG }, 08:43:50 DEBUG }, 08:43:50 DEBUG 1684568630, 08:43:50 DEBUG "value": [ 08:43:50 DEBUG "value": [ 08:43:50 DEBUG "0" 08:43:50 DEBUG 1684568630, 08:43:50 DEBUG 1684568630, 08:43:50 DEBUG ] 08:43:50 DEBUG "0" 08:43:50 DEBUG "0" 08:43:50 DEBUG } 08:43:50 DEBUG ] 08:43:50 DEBUG ] 08:43:50 DEBUG ] 08:43:50 DEBUG } 08:43:50 DEBUG } 08:43:50 DEBUG } 08:43:50 DEBUG ] 08:43:50 DEBUG ] 08:43:50 DEBUG } 08:43:50 DEBUG } 08:43:50 DEBUG } 08:43:50 DEBUG } 08:43:50 DEBUG } 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "pod": "ds-idrepo-2" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "pod": "ds-idrepo-1" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "0" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568631" 08:43:51 DEBUG --- status code --- 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:43:51 DEBUG { 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG { 08:43:51 DEBUG { 08:43:51 DEBUG "data": { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "data": { 08:43:51 DEBUG "result": [ 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG { 08:43:51 DEBUG "result": [ 08:43:51 DEBUG "result": [ 08:43:51 DEBUG "metric": { 08:43:51 DEBUG { 08:43:51 DEBUG { 08:43:51 DEBUG "pod": "ds-cts-2" 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG }, 08:43:51 DEBUG "pod": "ds-cts-2" 08:43:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:43:51 DEBUG "value": [ 08:43:51 DEBUG }, 08:43:51 DEBUG }, 08:43:51 DEBUG 08:43:51 DEBUG "value": [ 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "pod": "ds-idrepo-1" 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG }, 08:43:51 DEBUG "pod": "ds-idrepo-1" 08:43:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:43:51 DEBUG "value": [ 08:43:51 DEBUG }, 08:43:51 DEBUG }, 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG "value": [ 08:43:51 DEBUG "0" 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG ] 08:43:51 DEBUG "0" 08:43:51 DEBUG "0" 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG ] 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG ] 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG { 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG "data": { 08:43:51 DEBUG { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG "data": { 08:43:51 DEBUG { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "result": [ 08:43:51 DEBUG "instance": "10.142.0.17:9100" 08:43:51 DEBUG { 08:43:51 DEBUG }, 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "value": [ 08:43:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:43:51 DEBUG 08:43:51 DEBUG }, 08:43:51 DEBUG 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "instance": "10.142.0.61:9100" 08:43:51 DEBUG 08:43:51 DEBUG }, 08:43:51 DEBUG 08:43:51 DEBUG "value": [ 08:43:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG }, 08:43:51 DEBUG "0.00040000000000001514" 08:43:51 DEBUG "value": [ 08:43:51 DEBUG ] 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG } 08:43:51 DEBUG "0" 08:43:51 DEBUG ] 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568631" 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568631" 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "instance": "10.142.0.17:9100" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "instance": "10.142.0.61:9100" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "6602.666666666666" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "instance": "10.142.0.17:9100" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "instance": "10.142.0.61:9100" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "0" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "0" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "0.2" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568631" 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568631" 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "14444236800" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "instance": "10.142.0.17:9100" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "instance": "10.142.0.61:9100" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "2515.233333333333" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "pod": "ds-idrepo-1" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "0" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568631" 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "0" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "0" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:43:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568631" 08:43:51 INFO [http_cmd]: http status code OK 08:43:51 DEBUG --- status code --- 08:43:51 DEBUG http status code is 200 (expected 200) 08:43:51 DEBUG --- http response --- 08:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:43:51 DEBUG ----- output ----- 08:43:51 DEBUG { 08:43:51 DEBUG "status": "success", 08:43:51 DEBUG "data": { 08:43:51 DEBUG "resultType": "vector", 08:43:51 DEBUG "result": [ 08:43:51 DEBUG { 08:43:51 DEBUG "metric": { 08:43:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG [...] 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG 08:43:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:43:51 DEBUG }, 08:43:51 DEBUG "value": [ 08:43:51 DEBUG 1684568631, 08:43:51 DEBUG "1.031632127787338" 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG ] 08:43:51 DEBUG } 08:43:51 DEBUG } 08:44:27 INFO 08:44:27 INFO [loop_until]: kubectl --namespace=xlou top pods 08:44:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:44:27 INFO [loop_until]: OK (rc = 0) 08:44:27 DEBUG --- stdout --- 08:44:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 357m 9403Mi am-7849cf7bdb-lxfkl 323m 9342Mi am-7849cf7bdb-td7js 307m 9402Mi ds-cts-0 887m 13554Mi ds-cts-1 1219m 13634Mi ds-cts-2 644m 13684Mi ds-idrepo-0 232m 13662Mi ds-idrepo-1 12m 13555Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1013m 4384Mi 08:44:27 DEBUG --- stderr --- 08:44:27 DEBUG 08:44:39 INFO 08:44:39 INFO [loop_until]: kubectl --namespace=xlou top node 08:44:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:44:39 INFO [loop_until]: OK (rc = 0) 08:44:39 DEBUG --- stdout --- 08:44:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 389m 2% 11559Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4951Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 475m 2% 11676Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 414m 2% 12006Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 157m 0% 3132Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1365Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 98m 0% 5309Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14643Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 834m 5% 15011Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 290m 1% 14770Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 64m 0% 14575Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 1150m 7% 15021Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 1504m 9% 14828Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1048m 6% 6559Mi 11% 08:44:39 DEBUG --- stderr --- 08:44:39 DEBUG 08:44:50 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568690" 08:44:50 INFO [http_cmd]: http status code OK 08:44:50 DEBUG --- status code --- 08:44:50 DEBUG http status code is 200 (expected 200) 08:44:50 DEBUG --- http response --- 08:44:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:44:50 DEBUG ----- output ----- 08:44:50 DEBUG { 08:44:50 DEBUG "status": "success", 08:44:50 DEBUG "data": { 08:44:50 DEBUG "resultType": "vector", 08:44:50 DEBUG "result": [ 08:44:50 DEBUG { 08:44:50 DEBUG "metric": { 08:44:50 DEBUG "pod": "ds-idrepo-2" 08:44:50 DEBUG }, 08:44:50 DEBUG "value": [ 08:44:50 DEBUG 08:44:50 DEBUG 08:44:50 DEBUG 08:44:50 DEBUG [...] 08:44:50 DEBUG 08:44:50 DEBUG 08:44:50 DEBUG 08:44:50 DEBUG "pod": "ds-idrepo-1" 08:44:50 DEBUG }, 08:44:50 DEBUG "value": [ 08:44:50 DEBUG 1684568690, 08:44:50 DEBUG "0" 08:44:50 DEBUG ] 08:44:50 DEBUG } 08:44:50 DEBUG ] 08:44:50 DEBUG } 08:44:50 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "0" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG { 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG "data": { 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "data": { 08:44:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG }, 08:44:51 DEBUG "result": [ 08:44:51 DEBUG "value": [ 08:44:51 DEBUG { 08:44:51 DEBUG 08:44:51 DEBUG "metric": { 08:44:51 DEBUG 08:44:51 DEBUG "pod": "ds-cts-2" 08:44:51 DEBUG 08:44:51 DEBUG }, 08:44:51 DEBUG [...] 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "pod": "ds-idrepo-1" 08:44:51 DEBUG [...] 08:44:51 DEBUG }, 08:44:51 DEBUG 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG 08:44:51 DEBUG "8961.895785646964" 08:44:51 DEBUG "pod": "ds-idrepo-1" 08:44:51 DEBUG ] 08:44:51 DEBUG }, 08:44:51 DEBUG } 08:44:51 DEBUG "value": [ 08:44:51 DEBUG ] 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG } 08:44:51 DEBUG "0" 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "pod": "ds-idrepo-2" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "pod": "ds-idrepo-1" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "0" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "pod": "ds-cts-2" 08:44:51 DEBUG "pod": "ds-idrepo-2" 08:44:51 DEBUG }, 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "pod": "ds-idrepo-1" 08:44:51 DEBUG "pod": "ds-idrepo-1" 08:44:51 DEBUG }, 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "0" 08:44:51 DEBUG "0" 08:44:51 DEBUG ] 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568691" 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568691" 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG { 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG "status": "success", 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG "status": "success", 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG "data": { 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG "data": { 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG "result": [ 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG "result": [ 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:44:51 DEBUG { 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:44:51 DEBUG { 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG "metric": { 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG "metric": { 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "pod": "ds-cts-2" 08:44:51 DEBUG { 08:44:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG }, 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG }, 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "value": [ 08:44:51 DEBUG "data": { 08:44:51 DEBUG "value": [ 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG 08:44:51 DEBUG "result": [ 08:44:51 DEBUG 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG 08:44:51 DEBUG { 08:44:51 DEBUG 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG [...] 08:44:51 DEBUG "metric": { 08:44:51 DEBUG [...] 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "instance": "10.142.0.17:9100" 08:44:51 DEBUG 08:44:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:44:51 DEBUG 08:44:51 DEBUG "instance": "10.142.0.17:9100" 08:44:51 DEBUG }, 08:44:51 DEBUG 08:44:51 DEBUG }, 08:44:51 DEBUG 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG "pod": "ds-idrepo-1" 08:44:51 DEBUG 08:44:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG }, 08:44:51 DEBUG 08:44:51 DEBUG }, 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG [...] 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG "0" 08:44:51 DEBUG 08:44:51 DEBUG "0" 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG ] 08:44:51 DEBUG 08:44:51 DEBUG ] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG } 08:44:51 DEBUG 08:44:51 DEBUG } 08:44:51 DEBUG 08:44:51 DEBUG "instance": "10.142.0.61:9100" 08:44:51 DEBUG ] 08:44:51 DEBUG "pod": "ds-idrepo-1" 08:44:51 DEBUG ] 08:44:51 DEBUG "instance": "10.142.0.61:9100" 08:44:51 DEBUG }, 08:44:51 DEBUG } 08:44:51 DEBUG }, 08:44:51 DEBUG } 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG } 08:44:51 DEBUG "value": [ 08:44:51 DEBUG } 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "0" 08:44:51 DEBUG "0" 08:44:51 DEBUG "0.00040000000000001514" 08:44:51 DEBUG ] 08:44:51 DEBUG ] 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG ] 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568691" 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "instance": "10.142.0.17:9100" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "instance": "10.142.0.61:9100" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "6682.9" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "0" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "0.2" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "14409068544" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "instance": "10.142.0.17:9100" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "instance": "10.142.0.61:9100" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "3149.8999999999996" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "0" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "pod": "ds-idrepo-1" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "0.011953010636873979" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "pod": "ds-idrepo-1" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "0.9425070688030159" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:44:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568691" 08:44:51 INFO [http_cmd]: http status code OK 08:44:51 DEBUG --- status code --- 08:44:51 DEBUG http status code is 200 (expected 200) 08:44:51 DEBUG --- http response --- 08:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:44:51 DEBUG ----- output ----- 08:44:51 DEBUG { 08:44:51 DEBUG "status": "success", 08:44:51 DEBUG "data": { 08:44:51 DEBUG "resultType": "vector", 08:44:51 DEBUG "result": [ 08:44:51 DEBUG { 08:44:51 DEBUG "metric": { 08:44:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG [...] 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG 08:44:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:44:51 DEBUG }, 08:44:51 DEBUG "value": [ 08:44:51 DEBUG 1684568691, 08:44:51 DEBUG "1.029851152775636" 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG ] 08:44:51 DEBUG } 08:44:51 DEBUG } 08:45:27 INFO 08:45:27 INFO [loop_until]: kubectl --namespace=xlou top pods 08:45:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:45:28 INFO [loop_until]: OK (rc = 0) 08:45:28 DEBUG --- stdout --- 08:45:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 404m 9403Mi am-7849cf7bdb-lxfkl 298m 9342Mi am-7849cf7bdb-td7js 319m 9402Mi ds-cts-0 912m 13441Mi ds-cts-1 706m 13646Mi ds-cts-2 635m 13668Mi ds-idrepo-0 228m 13661Mi ds-idrepo-1 12m 13555Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 7m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1037m 4609Mi 08:45:28 DEBUG --- stderr --- 08:45:28 DEBUG 08:45:39 INFO 08:45:39 INFO [loop_until]: kubectl --namespace=xlou top node 08:45:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:45:40 INFO [loop_until]: OK (rc = 0) 08:45:40 DEBUG --- stdout --- 08:45:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 401m 2% 11558Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4950Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 474m 2% 11675Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 409m 2% 12009Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 156m 0% 3146Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 93m 0% 1358Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 69m 0% 14657Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 963m 6% 15028Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 290m 1% 14773Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 66m 0% 14578Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 671m 4% 15034Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 652m 4% 14817Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1082m 6% 6800Mi 11% 08:45:40 DEBUG --- stderr --- 08:45:40 DEBUG 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "ds-idrepo-2" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:45:51 DEBUG { 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG { 08:45:51 DEBUG "data": { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "result": [ 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG { 08:45:51 DEBUG "result": [ 08:45:51 DEBUG "metric": { 08:45:51 DEBUG { 08:45:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:45:51 DEBUG "metric": { 08:45:51 DEBUG }, 08:45:51 DEBUG "pod": "ds-cts-2" 08:45:51 DEBUG "value": [ 08:45:51 DEBUG }, 08:45:51 DEBUG 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG 08:45:51 DEBUG }, 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG "value": [ 08:45:51 DEBUG }, 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG "395380135.0328123" 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG ] 08:45:51 DEBUG "0" 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "ds-idrepo-2" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "ds-idrepo-2" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "ds-cts-2" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:45:51 DEBUG { 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG { 08:45:51 DEBUG "data": { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "result": [ 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG { 08:45:51 DEBUG "result": [ 08:45:51 DEBUG "metric": { 08:45:51 DEBUG { 08:45:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:45:51 DEBUG "metric": { 08:45:51 DEBUG }, 08:45:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:45:51 DEBUG "value": [ 08:45:51 DEBUG }, 08:45:51 DEBUG 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:45:51 DEBUG 08:45:51 DEBUG }, 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG "value": [ 08:45:51 DEBUG }, 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG "0" 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG ] 08:45:51 DEBUG "0" 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568751" 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568751" 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG { 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG "data": { 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "data": { 08:45:51 DEBUG "pod": "ds-cts-2" 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG }, 08:45:51 DEBUG "result": [ 08:45:51 DEBUG "value": [ 08:45:51 DEBUG { 08:45:51 DEBUG 08:45:51 DEBUG "metric": { 08:45:51 DEBUG 08:45:51 DEBUG "instance": "10.142.0.17:9100" 08:45:51 DEBUG 08:45:51 DEBUG }, 08:45:51 DEBUG [...] 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG [...] 08:45:51 DEBUG }, 08:45:51 DEBUG 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG 08:45:51 DEBUG "0" 08:45:51 DEBUG "instance": "10.142.0.61:9100" 08:45:51 DEBUG ] 08:45:51 DEBUG }, 08:45:51 DEBUG } 08:45:51 DEBUG "value": [ 08:45:51 DEBUG ] 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG } 08:45:51 DEBUG "0" 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG ] 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG } 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG { 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG "data": { 08:45:51 DEBUG { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG "data": { 08:45:51 DEBUG { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "result": [ 08:45:51 DEBUG "instance": "10.142.0.17:9100" 08:45:51 DEBUG { 08:45:51 DEBUG }, 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "value": [ 08:45:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:45:51 DEBUG 08:45:51 DEBUG }, 08:45:51 DEBUG 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "instance": "10.142.0.61:9100" 08:45:51 DEBUG 08:45:51 DEBUG }, 08:45:51 DEBUG 08:45:51 DEBUG "value": [ 08:45:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG }, 08:45:51 DEBUG "0.0001999999999990602" 08:45:51 DEBUG "value": [ 08:45:51 DEBUG ] 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG } 08:45:51 DEBUG "14423007232" 08:45:51 DEBUG ] 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568751" 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "instance": "10.142.0.17:9100" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "instance": "10.142.0.61:9100" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "6700.666666666667" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0.2" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "instance": "10.142.0.17:9100" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "instance": "10.142.0.61:9100" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "2816.3333333333335" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0.008353135082657325" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "pod": "ds-idrepo-1" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "0.6347342827370615" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:45:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568751" 08:45:51 INFO [http_cmd]: http status code OK 08:45:51 DEBUG --- status code --- 08:45:51 DEBUG http status code is 200 (expected 200) 08:45:51 DEBUG --- http response --- 08:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:45:51 DEBUG ----- output ----- 08:45:51 DEBUG { 08:45:51 DEBUG "status": "success", 08:45:51 DEBUG "data": { 08:45:51 DEBUG "resultType": "vector", 08:45:51 DEBUG "result": [ 08:45:51 DEBUG { 08:45:51 DEBUG "metric": { 08:45:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG [...] 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG 08:45:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:45:51 DEBUG }, 08:45:51 DEBUG "value": [ 08:45:51 DEBUG 1684568751, 08:45:51 DEBUG "1.0317211440583252" 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG ] 08:45:51 DEBUG } 08:45:51 DEBUG } 08:46:28 INFO 08:46:28 INFO [loop_until]: kubectl --namespace=xlou top pods 08:46:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:46:28 INFO [loop_until]: OK (rc = 0) 08:46:28 DEBUG --- stdout --- 08:46:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 377m 9403Mi am-7849cf7bdb-lxfkl 287m 9342Mi am-7849cf7bdb-td7js 312m 9402Mi ds-cts-0 858m 13481Mi ds-cts-1 664m 13671Mi ds-cts-2 882m 13677Mi ds-idrepo-0 238m 13661Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 7m 68Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 974m 4886Mi 08:46:28 DEBUG --- stderr --- 08:46:28 DEBUG 08:46:40 INFO 08:46:40 INFO [loop_until]: kubectl --namespace=xlou top node 08:46:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:46:40 INFO [loop_until]: OK (rc = 0) 08:46:40 DEBUG --- stdout --- 08:46:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 387m 2% 11557Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 81m 0% 4948Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 481m 3% 11674Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 427m 2% 12008Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 163m 1% 3150Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 90m 0% 1363Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 97m 0% 5298Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14645Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 716m 4% 15060Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 277m 1% 14772Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14576Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 686m 4% 15039Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 677m 4% 14830Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 761m 4% 6603Mi 11% 08:46:40 DEBUG --- stderr --- 08:46:40 DEBUG 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "ds-idrepo-2" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "ds-idrepo-1" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "8433.979604887858" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "ds-cts-2" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "ds-idrepo-1" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "ds-idrepo-2" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "ds-idrepo-1" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "ds-idrepo-2" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "ds-idrepo-1" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "ds-cts-2" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "ds-idrepo-1" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568811" 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568811" 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG ----- output ----- 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG { 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:46:51 DEBUG { 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG "data": { 08:46:51 DEBUG { 08:46:51 DEBUG { 08:46:51 DEBUG "data": { 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:46:51 DEBUG "result": [ 08:46:51 DEBUG "data": { 08:46:51 DEBUG "data": { 08:46:51 DEBUG "result": [ 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG { 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "result": [ 08:46:51 DEBUG "result": [ 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "pod": "ds-cts-2" 08:46:51 DEBUG { 08:46:51 DEBUG { 08:46:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:46:51 DEBUG "data": { 08:46:51 DEBUG }, 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG }, 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "value": [ 08:46:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:46:51 DEBUG "instance": "10.142.0.17:9100" 08:46:51 DEBUG "value": [ 08:46:51 DEBUG "result": [ 08:46:51 DEBUG 08:46:51 DEBUG }, 08:46:51 DEBUG }, 08:46:51 DEBUG 08:46:51 DEBUG { 08:46:51 DEBUG 08:46:51 DEBUG "value": [ 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG "metric": { 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "instance": "10.142.0.17:9100" 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG }, 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "ds-idrepo-1" 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:46:51 DEBUG 08:46:51 DEBUG }, 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG }, 08:46:51 DEBUG [...] 08:46:51 DEBUG "value": [ 08:46:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:46:51 DEBUG "instance": "10.142.0.61:9100" 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG }, 08:46:51 DEBUG }, 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG 08:46:51 DEBUG "0" 08:46:51 DEBUG "value": [ 08:46:51 DEBUG "value": [ 08:46:51 DEBUG "0.2" 08:46:51 DEBUG 08:46:51 DEBUG ] 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG ] 08:46:51 DEBUG "instance": "10.142.0.61:9100" 08:46:51 DEBUG } 08:46:51 DEBUG "14428618752" 08:46:51 DEBUG "0.0003333333333333262" 08:46:51 DEBUG } 08:46:51 DEBUG }, 08:46:51 DEBUG ] 08:46:51 DEBUG ] 08:46:51 DEBUG ] 08:46:51 DEBUG ] 08:46:51 DEBUG "value": [ 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG "0.0007000000000005001" 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568811" 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "instance": "10.142.0.17:9100" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "instance": "10.142.0.61:9100" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "6969.1" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "instance": "10.142.0.17:9100" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "instance": "10.142.0.61:9100" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "3271.0333333333333" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "1.637765530731536" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "0" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 08:46:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568811" 08:46:51 INFO [http_cmd]: http status code OK 08:46:51 DEBUG --- status code --- 08:46:51 DEBUG http status code is 200 (expected 200) 08:46:51 DEBUG --- http response --- 08:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:46:51 DEBUG ----- output ----- 08:46:51 DEBUG { 08:46:51 DEBUG "status": "success", 08:46:51 DEBUG "data": { 08:46:51 DEBUG "resultType": "vector", 08:46:51 DEBUG "result": [ 08:46:51 DEBUG { 08:46:51 DEBUG "metric": { 08:46:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG [...] 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG 08:46:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:46:51 DEBUG }, 08:46:51 DEBUG "value": [ 08:46:51 DEBUG 1684568811, 08:46:51 DEBUG "1.031224955733517" 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG ] 08:46:51 DEBUG } 08:46:51 DEBUG } 127.0.0.1 - - [20/May/2023 08:46:53] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:47:00] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:47:07] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:47:13] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:47:20] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:47:27] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:47:28 INFO 08:47:28 INFO [loop_until]: kubectl --namespace=xlou top pods 08:47:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:47:28 INFO [loop_until]: OK (rc = 0) 08:47:28 DEBUG --- stdout --- 08:47:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 346m 9403Mi am-7849cf7bdb-lxfkl 309m 9342Mi am-7849cf7bdb-td7js 325m 9402Mi ds-cts-0 594m 13467Mi ds-cts-1 1315m 13575Mi ds-cts-2 597m 13688Mi ds-idrepo-0 238m 13662Mi ds-idrepo-1 13m 13554Mi ds-idrepo-2 16m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 24m 71Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 632m 4895Mi 08:47:28 DEBUG --- stderr --- 08:47:28 DEBUG 127.0.0.1 - - [20/May/2023 08:47:33] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:47:40 INFO 08:47:40 INFO [loop_until]: kubectl --namespace=xlou top node 08:47:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:47:40 INFO [loop_until]: OK (rc = 0) 08:47:40 DEBUG --- stdout --- 08:47:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 390m 2% 11557Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4951Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 470m 2% 11673Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 393m 2% 12007Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 156m 0% 3113Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 130m 0% 1368Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 97m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14648Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 963m 6% 14950Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 292m 1% 14772Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14576Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 1553m 9% 14933Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 1351m 8% 14810Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1120m 7% 7056Mi 12% 08:47:40 DEBUG --- stderr --- 08:47:40 DEBUG 127.0.0.1 - - [20/May/2023 08:47:40] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:47:47] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "ds-idrepo-2" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "ds-idrepo-1" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG { 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG "data": { 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "data": { 08:47:51 DEBUG "pod": "ds-cts-2" 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG }, 08:47:51 DEBUG "result": [ 08:47:51 DEBUG "value": [ 08:47:51 DEBUG { 08:47:51 DEBUG 08:47:51 DEBUG "metric": { 08:47:51 DEBUG 08:47:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:47:51 DEBUG 08:47:51 DEBUG }, 08:47:51 DEBUG [...] 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "ds-idrepo-1" 08:47:51 DEBUG [...] 08:47:51 DEBUG }, 08:47:51 DEBUG 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG 08:47:51 DEBUG "0" 08:47:51 DEBUG "pod": "ds-idrepo-1" 08:47:51 DEBUG ] 08:47:51 DEBUG }, 08:47:51 DEBUG } 08:47:51 DEBUG "value": [ 08:47:51 DEBUG ] 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG } 08:47:51 DEBUG "0" 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "ds-idrepo-2" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "ds-idrepo-1" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "ds-idrepo-2" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "ds-idrepo-1" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "ds-idrepo-0" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "ds-cts-2" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "ds-idrepo-1" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0.2" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568871" 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568871" 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "ds-cts-2" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "ds-idrepo-1" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG } 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG } 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:47:51 DEBUG }, 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568871" 08:47:51 DEBUG "value": [ 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG 1684568871, 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG "14317449216" 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG ] 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG } 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:47:51 DEBUG ] 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG } 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG } 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "instance": "10.142.0.17:9100" 08:47:51 DEBUG "instance": "10.142.0.17:9100" 08:47:51 DEBUG }, 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "instance": "10.142.0.61:9100" 08:47:51 DEBUG "instance": "10.142.0.61:9100" 08:47:51 DEBUG }, 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0.9118000000000004" 08:47:51 DEBUG "0.003999999999999204" 08:47:51 DEBUG ] 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:47:51 DEBUG "instance": "10.142.0.17:9100" 08:47:51 DEBUG }, 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:47:51 DEBUG "instance": "10.142.0.61:9100" 08:47:51 DEBUG }, 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG "6830.766666666666" 08:47:51 DEBUG ] 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "instance": "10.142.0.17:9100" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "instance": "10.142.0.61:9100" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "5231.200000000001" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "ds-idrepo-1" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "pod": "ds-idrepo-0" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "0.5080096183154401" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 08:47:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568871" 08:47:51 INFO [http_cmd]: http status code OK 08:47:51 DEBUG --- status code --- 08:47:51 DEBUG http status code is 200 (expected 200) 08:47:51 DEBUG --- http response --- 08:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:47:51 DEBUG ----- output ----- 08:47:51 DEBUG { 08:47:51 DEBUG "status": "success", 08:47:51 DEBUG "data": { 08:47:51 DEBUG "resultType": "vector", 08:47:51 DEBUG "result": [ 08:47:51 DEBUG { 08:47:51 DEBUG "metric": { 08:47:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG [...] 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG 08:47:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:47:51 DEBUG }, 08:47:51 DEBUG "value": [ 08:47:51 DEBUG 1684568871, 08:47:51 DEBUG "1.1039976628654096" 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG ] 08:47:51 DEBUG } 08:47:51 DEBUG } 127.0.0.1 - - [20/May/2023 08:47:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:48:00] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:48:07] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:48:14] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:48:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:48:28] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:48:28 INFO 08:48:28 INFO [loop_until]: kubectl --namespace=xlou top pods 08:48:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:48:28 INFO [loop_until]: OK (rc = 0) 08:48:28 DEBUG --- stdout --- 08:48:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 339m 9403Mi am-7849cf7bdb-lxfkl 298m 9342Mi am-7849cf7bdb-td7js 321m 9403Mi ds-cts-0 644m 13442Mi ds-cts-1 1146m 13495Mi ds-cts-2 863m 13595Mi ds-idrepo-0 227m 13661Mi ds-idrepo-1 12m 13555Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 39m 75Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 994m 4938Mi 08:48:28 DEBUG --- stderr --- 08:48:28 DEBUG 127.0.0.1 - - [20/May/2023 08:48:34] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:48:40 INFO 08:48:40 INFO [loop_until]: kubectl --namespace=xlou top node 08:48:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:48:40 INFO [loop_until]: OK (rc = 0) 08:48:40 DEBUG --- stdout --- 08:48:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 402m 2% 11559Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 4949Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 476m 2% 11675Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 420m 2% 12006Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 176m 1% 3091Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 135m 0% 1370Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 98m 0% 5300Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14646Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 733m 4% 14874Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 294m 1% 14771Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 69m 0% 14577Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 809m 5% 14933Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 879m 5% 14846Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1079m 6% 7079Mi 12% 08:48:40 DEBUG --- stderr --- 08:48:40 DEBUG 127.0.0.1 - - [20/May/2023 08:48:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:48:48] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG { 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "result": [ 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG { 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "ds-cts-2" 08:48:51 DEBUG }, 08:48:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:48:51 DEBUG "pod": "ds-idrepo-2" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG }, 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG "value": [ 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG }, 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG }, 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG "418260285.0400396" 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "ds-idrepo-2" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "ds-idrepo-2" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "ds-cts-2" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "ds-cts-2" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0.2" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "instance": "10.142.0.17:9100" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "instance": "10.142.0.61:9100" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "6631.233333333334" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "instance": "10.142.0.17:9100" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "instance": "10.142.0.61:9100" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0.08503333333333292" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568931" 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568931" 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "14318694400" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "instance": "10.142.0.17:9100" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "instance": "10.142.0.61:9100" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "instance": "10.142.0.17:9100" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "instance": "10.142.0.61:9100" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "3008.6000000000004" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "0" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG { 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:48:51 DEBUG "data": { 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG { 08:48:51 DEBUG "result": [ 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG { 08:48:51 DEBUG "data": { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:48:51 DEBUG "result": [ 08:48:51 DEBUG }, 08:48:51 DEBUG { 08:48:51 DEBUG "value": [ 08:48:51 DEBUG "metric": { 08:48:51 DEBUG 08:48:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:48:51 DEBUG 08:48:51 DEBUG }, 08:48:51 DEBUG 08:48:51 DEBUG "value": [ 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG 08:48:51 DEBUG }, 08:48:51 DEBUG 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "pod": "ds-idrepo-1" 08:48:51 DEBUG "0.005113776478476491" 08:48:51 DEBUG }, 08:48:51 DEBUG ] 08:48:51 DEBUG "value": [ 08:48:51 DEBUG } 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG ] 08:48:51 DEBUG "0.4476376940571542" 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 08:48:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568931" 08:48:51 INFO [http_cmd]: http status code OK 08:48:51 DEBUG --- status code --- 08:48:51 DEBUG http status code is 200 (expected 200) 08:48:51 DEBUG --- http response --- 08:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:48:51 DEBUG ----- output ----- 08:48:51 DEBUG { 08:48:51 DEBUG "status": "success", 08:48:51 DEBUG "data": { 08:48:51 DEBUG "resultType": "vector", 08:48:51 DEBUG "result": [ 08:48:51 DEBUG { 08:48:51 DEBUG "metric": { 08:48:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG [...] 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG 08:48:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:48:51 DEBUG }, 08:48:51 DEBUG "value": [ 08:48:51 DEBUG 1684568931, 08:48:51 DEBUG "1.0354246891217316" 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG ] 08:48:51 DEBUG } 08:48:51 DEBUG } 127.0.0.1 - - [20/May/2023 08:48:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:49:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:49:08] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:49:14] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:49:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:49:28] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:49:28 INFO 08:49:28 INFO [loop_until]: kubectl --namespace=xlou top pods 08:49:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:49:28 INFO [loop_until]: OK (rc = 0) 08:49:28 DEBUG --- stdout --- 08:49:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 696m 9403Mi am-7849cf7bdb-lxfkl 302m 9342Mi am-7849cf7bdb-td7js 316m 9403Mi ds-cts-0 650m 13456Mi ds-cts-1 671m 13507Mi ds-cts-2 889m 13607Mi ds-idrepo-0 239m 13662Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 42m 75Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1038m 4962Mi 08:49:28 DEBUG --- stderr --- 08:49:28 DEBUG 127.0.0.1 - - [20/May/2023 08:49:34] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:49:40 INFO 08:49:40 INFO [loop_until]: kubectl --namespace=xlou top node 08:49:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:49:40 INFO [loop_until]: OK (rc = 0) 08:49:40 DEBUG --- stdout --- 08:49:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 395m 2% 11557Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 76m 0% 4948Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 476m 2% 11669Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 426m 2% 12013Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 158m 0% 3082Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 136m 0% 1371Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 100m 0% 5298Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 63m 0% 14649Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 977m 6% 14891Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 291m 1% 14775Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 67m 0% 14582Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 718m 4% 14945Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 1329m 8% 14822Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1085m 6% 7096Mi 12% 08:49:40 DEBUG --- stderr --- 08:49:40 DEBUG 127.0.0.1 - - [20/May/2023 08:49:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:49:48] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG { 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "result": [ 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG { 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "ds-idrepo-2" 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:49:51 DEBUG }, 08:49:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:49:51 DEBUG "pod": "ds-cts-2" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG }, 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG "value": [ 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:49:51 DEBUG }, 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG }, 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG "417104885.60677415" 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "ds-idrepo-2" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "ds-idrepo-2" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "ds-cts-2" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "ds-cts-2" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568991" 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0.2" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "instance": "10.142.0.17:9100" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "instance": "10.142.0.61:9100" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0.023199999999998985" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "instance": "10.142.0.17:9100" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "instance": "10.142.0.61:9100" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "6730.566666666667" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684568991" 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "14331117568" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "instance": "10.142.0.17:9100" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "instance": "10.142.0.61:9100" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684568991" 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG "data": { 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG "result": [ 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG { 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:49:51 DEBUG "metric": { 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:49:51 DEBUG { 08:49:51 DEBUG }, 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "value": [ 08:49:51 DEBUG "data": { 08:49:51 DEBUG 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG 08:49:51 DEBUG "result": [ 08:49:51 DEBUG 08:49:51 DEBUG { 08:49:51 DEBUG [...] 08:49:51 DEBUG "metric": { 08:49:51 DEBUG 08:49:51 DEBUG "instance": "10.142.0.17:9100" 08:49:51 DEBUG 08:49:51 DEBUG }, 08:49:51 DEBUG 08:49:51 DEBUG "value": [ 08:49:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:49:51 DEBUG 08:49:51 DEBUG }, 08:49:51 DEBUG 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG [...] 08:49:51 DEBUG "0" 08:49:51 DEBUG 08:49:51 DEBUG ] 08:49:51 DEBUG 08:49:51 DEBUG } 08:49:51 DEBUG 08:49:51 DEBUG ] 08:49:51 DEBUG "instance": "10.142.0.61:9100" 08:49:51 DEBUG } 08:49:51 DEBUG }, 08:49:51 DEBUG } 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "2925.8333333333335" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:49:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:49:51 DEBUG }, 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG "pod": "ds-idrepo-1" 08:49:51 DEBUG }, 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "0.4742982252638905" 08:49:51 DEBUG "0.005286506954370826" 08:49:51 DEBUG ] 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 DEBUG } 08:49:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684568991" 08:49:51 INFO [http_cmd]: http status code OK 08:49:51 DEBUG --- status code --- 08:49:51 DEBUG http status code is 200 (expected 200) 08:49:51 DEBUG --- http response --- 08:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:49:51 DEBUG ----- output ----- 08:49:51 DEBUG { 08:49:51 DEBUG "status": "success", 08:49:51 DEBUG "data": { 08:49:51 DEBUG "resultType": "vector", 08:49:51 DEBUG "result": [ 08:49:51 DEBUG { 08:49:51 DEBUG "metric": { 08:49:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG [...] 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG 08:49:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:49:51 DEBUG }, 08:49:51 DEBUG "value": [ 08:49:51 DEBUG 1684568991, 08:49:51 DEBUG "1.029072036827006" 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG ] 08:49:51 DEBUG } 08:49:51 DEBUG } 127.0.0.1 - - [20/May/2023 08:49:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:50:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:50:08] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:50:15] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:50:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:50:28] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:50:28 INFO 08:50:28 INFO [loop_until]: kubectl --namespace=xlou top pods 08:50:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:50:28 INFO [loop_until]: OK (rc = 0) 08:50:28 DEBUG --- stdout --- 08:50:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 377m 9403Mi am-7849cf7bdb-lxfkl 325m 9342Mi am-7849cf7bdb-td7js 308m 9407Mi ds-cts-0 630m 13454Mi ds-cts-1 1157m 13413Mi ds-cts-2 667m 13595Mi ds-idrepo-0 239m 13662Mi ds-idrepo-1 12m 13555Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 7m 3297Mi lodemon-58bb5f9bf4-7mmfr 41m 76Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1002m 4972Mi 08:50:28 DEBUG --- stderr --- 08:50:28 DEBUG 127.0.0.1 - - [20/May/2023 08:50:35] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:50:40 INFO 08:50:40 INFO [loop_until]: kubectl --namespace=xlou top node 08:50:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:50:40 INFO [loop_until]: OK (rc = 0) 08:50:40 DEBUG --- stdout --- 08:50:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 368m 2% 11557Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 4948Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 506m 3% 11673Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 418m 2% 12010Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 160m 1% 3132Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 132m 0% 1373Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 97m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 65m 0% 14648Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 827m 5% 14799Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 289m 1% 14775Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 69m 0% 14580Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 931m 5% 14990Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 912m 5% 14858Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1080m 6% 7105Mi 12% 08:50:40 DEBUG --- stderr --- 08:50:40 DEBUG 127.0.0.1 - - [20/May/2023 08:50:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:50:48] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "134051195.52447553" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "ds-cts-2" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "ds-idrepo-1" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:50:51 DEBUG { 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG { 08:50:51 DEBUG { 08:50:51 DEBUG "data": { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "data": { 08:50:51 DEBUG "result": [ 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG { 08:50:51 DEBUG "result": [ 08:50:51 DEBUG "result": [ 08:50:51 DEBUG "metric": { 08:50:51 DEBUG { 08:50:51 DEBUG { 08:50:51 DEBUG "pod": "ds-idrepo-2" 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG }, 08:50:51 DEBUG "pod": "ds-idrepo-2" 08:50:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:50:51 DEBUG "value": [ 08:50:51 DEBUG }, 08:50:51 DEBUG }, 08:50:51 DEBUG 08:50:51 DEBUG "value": [ 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "ds-idrepo-1" 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG }, 08:50:51 DEBUG "pod": "ds-idrepo-1" 08:50:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:50:51 DEBUG "value": [ 08:50:51 DEBUG }, 08:50:51 DEBUG }, 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG "value": [ 08:50:51 DEBUG "0" 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG ] 08:50:51 DEBUG "0" 08:50:51 DEBUG "0" 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG ] 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "ds-idrepo-2" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "ds-idrepo-1" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "ds-cts-2" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG "pod": "ds-idrepo-1" 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG }, 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG "value": [ 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG "0" 08:50:51 DEBUG { 08:50:51 DEBUG ] 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG } 08:50:51 DEBUG "data": { 08:50:51 DEBUG ] 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG } 08:50:51 DEBUG "result": [ 08:50:51 DEBUG } 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "instance": "10.142.0.17:9100" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "instance": "10.142.0.61:9100" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "6589.166666666666" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "ds-cts-2" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "ds-idrepo-1" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0.2" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "instance": "10.142.0.17:9100" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "instance": "10.142.0.61:9100" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0.00026666666666793997" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569051" 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "14375124992" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "instance": "10.142.0.17:9100" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "instance": "10.142.0.61:9100" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "instance": "10.142.0.17:9100" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "instance": "10.142.0.61:9100" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "2831.2666666666664" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG } 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:50:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:50:51 DEBUG }, 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:50:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:50:51 DEBUG }, 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "0.005100414510489363" 08:50:51 DEBUG "0" 08:50:51 DEBUG ] 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 DEBUG } 08:50:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569051" 08:50:51 INFO [http_cmd]: http status code OK 08:50:51 DEBUG --- status code --- 08:50:51 DEBUG http status code is 200 (expected 200) 08:50:51 DEBUG --- http response --- 08:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:50:51 DEBUG ----- output ----- 08:50:51 DEBUG { 08:50:51 DEBUG "status": "success", 08:50:51 DEBUG "data": { 08:50:51 DEBUG "resultType": "vector", 08:50:51 DEBUG "result": [ 08:50:51 DEBUG { 08:50:51 DEBUG "metric": { 08:50:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG [...] 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG 08:50:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:50:51 DEBUG }, 08:50:51 DEBUG "value": [ 08:50:51 DEBUG 1684569051, 08:50:51 DEBUG "1.0258581982464494" 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG ] 08:50:51 DEBUG } 08:50:51 DEBUG } 127.0.0.1 - - [20/May/2023 08:50:55] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:51:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:51:08] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:51:14] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:51:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:51:28] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:51:28 INFO 08:51:28 INFO [loop_until]: kubectl --namespace=xlou top pods 08:51:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:51:28 INFO [loop_until]: OK (rc = 0) 08:51:28 DEBUG --- stdout --- 08:51:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 370m 9404Mi am-7849cf7bdb-lxfkl 292m 9342Mi am-7849cf7bdb-td7js 322m 9402Mi ds-cts-0 655m 13466Mi ds-cts-1 691m 13424Mi ds-cts-2 686m 13609Mi ds-idrepo-0 240m 13662Mi ds-idrepo-1 11m 13555Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 37m 76Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1004m 4980Mi 08:51:28 DEBUG --- stderr --- 08:51:28 DEBUG 127.0.0.1 - - [20/May/2023 08:51:34] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:51:40 INFO 08:51:40 INFO [loop_until]: kubectl --namespace=xlou top node 08:51:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:51:40 INFO [loop_until]: OK (rc = 0) 08:51:40 DEBUG --- stdout --- 08:51:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 395m 2% 11557Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 81m 0% 4951Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 475m 2% 11676Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 396m 2% 12006Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 160m 1% 3134Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 135m 0% 1365Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 93m 0% 5300Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14648Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 953m 5% 14825Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 291m 1% 14775Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14580Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 1473m 9% 14882Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 1275m 8% 14815Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1043m 6% 7111Mi 12% 08:51:40 DEBUG --- stderr --- 08:51:40 DEBUG 127.0.0.1 - - [20/May/2023 08:51:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:51:48] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "219876047.3842491" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "ds-idrepo-2" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "ds-idrepo-1" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "ds-cts-2" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "ds-idrepo-1" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "ds-idrepo-2" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "ds-idrepo-1" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "ds-idrepo-2" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "ds-idrepo-1" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "ds-cts-2" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "ds-idrepo-1" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "ds-cts-2" 08:51:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG }, 08:51:51 DEBUG }, 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG "value": [ 08:51:51 DEBUG "value": [ 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG [...] 08:51:51 DEBUG [...] 08:51:51 DEBUG { 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "data": { 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "pod": "ds-idrepo-1" 08:51:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:51:51 DEBUG "result": [ 08:51:51 DEBUG }, 08:51:51 DEBUG }, 08:51:51 DEBUG { 08:51:51 DEBUG "value": [ 08:51:51 DEBUG "value": [ 08:51:51 DEBUG "metric": { 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "instance": "10.142.0.17:9100" 08:51:51 DEBUG "0" 08:51:51 DEBUG "0.2" 08:51:51 DEBUG }, 08:51:51 DEBUG ] 08:51:51 DEBUG ] 08:51:51 DEBUG "value": [ 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 DEBUG 08:51:51 DEBUG ] 08:51:51 DEBUG ] 08:51:51 DEBUG 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 DEBUG 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "instance": "10.142.0.61:9100" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "6739" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "instance": "10.142.0.17:9100" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "instance": "10.142.0.61:9100" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0.0003333333333330302" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "14263476224" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569111" 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "instance": "10.142.0.17:9100" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "instance": "10.142.0.61:9100" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "instance": "10.142.0.17:9100" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "instance": "10.142.0.61:9100" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "3079.6" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "0.504368851947604" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 08:51:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569111" 08:51:51 INFO [http_cmd]: http status code OK 08:51:51 DEBUG --- status code --- 08:51:51 DEBUG http status code is 200 (expected 200) 08:51:51 DEBUG --- http response --- 08:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:51:51 DEBUG ----- output ----- 08:51:51 DEBUG { 08:51:51 DEBUG "status": "success", 08:51:51 DEBUG "data": { 08:51:51 DEBUG "resultType": "vector", 08:51:51 DEBUG "result": [ 08:51:51 DEBUG { 08:51:51 DEBUG "metric": { 08:51:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG [...] 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG 08:51:51 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:51:51 DEBUG }, 08:51:51 DEBUG "value": [ 08:51:51 DEBUG 1684569111, 08:51:51 DEBUG "1.0338310600134923" 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG ] 08:51:51 DEBUG } 08:51:51 DEBUG } 127.0.0.1 - - [20/May/2023 08:51:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:52:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:52:08] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:52:14] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:52:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:52:28] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:52:28 INFO 08:52:28 INFO [loop_until]: kubectl --namespace=xlou top pods 08:52:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:52:28 INFO [loop_until]: OK (rc = 0) 08:52:28 DEBUG --- stdout --- 08:52:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 352m 9404Mi am-7849cf7bdb-lxfkl 300m 9342Mi am-7849cf7bdb-td7js 310m 9402Mi ds-cts-0 634m 13422Mi ds-cts-1 692m 13438Mi ds-cts-2 639m 13516Mi ds-idrepo-0 231m 13662Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 48m 76Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 989m 4989Mi 08:52:28 DEBUG --- stderr --- 08:52:28 DEBUG 127.0.0.1 - - [20/May/2023 08:52:34] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:52:40 INFO 08:52:40 INFO [loop_until]: kubectl --namespace=xlou top node 08:52:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:52:40 INFO [loop_until]: OK (rc = 0) 08:52:40 DEBUG --- stdout --- 08:52:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 382m 2% 11561Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4949Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 473m 2% 11677Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 412m 2% 12011Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 161m 1% 3152Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 129m 0% 1367Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 63m 0% 14646Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1608m 10% 14770Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 292m 1% 14774Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 70m 0% 14578Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 665m 4% 14876Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 654m 4% 14801Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1080m 6% 7122Mi 12% 08:52:40 DEBUG --- stderr --- 08:52:40 DEBUG 127.0.0.1 - - [20/May/2023 08:52:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:52:48] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "ds-cts-0" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "786094.7479945513" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "ds-idrepo-2" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "ds-idrepo-1" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "ds-cts-2" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "ds-idrepo-1" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "ds-idrepo-2" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "ds-idrepo-1" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "ds-cts-0" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "5.0552444377175725" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "ds-idrepo-2" 08:52:51 DEBUG "pod": "ds-cts-2" 08:52:51 DEBUG }, 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "ds-idrepo-1" 08:52:51 DEBUG "pod": "ds-idrepo-1" 08:52:51 DEBUG }, 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0" 08:52:51 DEBUG "0" 08:52:51 DEBUG ] 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569171" 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG }, 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG "value": [ 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG "0.2" 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:52:51 DEBUG ] 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG } 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:52:51 DEBUG { 08:52:51 DEBUG ] 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG } 08:52:51 DEBUG { 08:52:51 DEBUG "data": { 08:52:51 DEBUG } 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "result": [ 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG { 08:52:51 DEBUG "result": [ 08:52:51 DEBUG "metric": { 08:52:51 DEBUG { 08:52:51 DEBUG "pod": "ds-cts-2" 08:52:51 DEBUG "metric": { 08:52:51 DEBUG }, 08:52:51 DEBUG "instance": "10.142.0.17:9100" 08:52:51 DEBUG "value": [ 08:52:51 DEBUG }, 08:52:51 DEBUG 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG 08:52:51 DEBUG "pod": "ds-idrepo-1" 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG 08:52:51 DEBUG }, 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG "instance": "10.142.0.61:9100" 08:52:51 DEBUG "value": [ 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG }, 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:52:51 DEBUG "value": [ 08:52:51 DEBUG "0" 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG ] 08:52:51 DEBUG { 08:52:51 DEBUG "6735.866666666667" 08:52:51 DEBUG } 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG ] 08:52:51 DEBUG ] 08:52:51 DEBUG "data": { 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG "result": [ 08:52:51 DEBUG } 08:52:51 DEBUG { 08:52:51 DEBUG } 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "instance": "10.142.0.17:9100" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "instance": "10.142.0.61:9100" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0.00036666666666747006" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "14262538240" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "instance": "10.142.0.17:9100" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "instance": "10.142.0.61:9100" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "instance": "10.142.0.17:9100" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "instance": "10.142.0.61:9100" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "4972.133333333333" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "ds-cts-0" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "4.691993340396549" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569171" 08:52:51 INFO [http_cmd]: http status code OK 08:52:51 DEBUG --- status code --- 08:52:51 DEBUG http status code is 200 (expected 200) 08:52:51 DEBUG --- http response --- 08:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:52:51 DEBUG ----- output ----- 08:52:51 DEBUG { 08:52:51 DEBUG "status": "success", 08:52:51 DEBUG "data": { 08:52:51 DEBUG "resultType": "vector", 08:52:51 DEBUG "result": [ 08:52:51 DEBUG { 08:52:51 DEBUG "metric": { 08:52:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG [...] 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG 08:52:51 DEBUG "pod": "ds-cts-0" 08:52:51 DEBUG }, 08:52:51 DEBUG "value": [ 08:52:51 DEBUG 1684569171, 08:52:51 DEBUG "0.5957018675337431" 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG ] 08:52:51 DEBUG } 08:52:51 DEBUG } 08:52:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569172" 08:52:52 INFO [http_cmd]: http status code OK 08:52:52 DEBUG --- status code --- 08:52:52 DEBUG http status code is 200 (expected 200) 08:52:52 DEBUG --- http response --- 08:52:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:52:52 DEBUG ----- output ----- 08:52:52 DEBUG { 08:52:52 DEBUG "status": "success", 08:52:52 DEBUG "data": { 08:52:52 DEBUG "resultType": "vector", 08:52:52 DEBUG "result": [ 08:52:52 DEBUG { 08:52:52 DEBUG "metric": { 08:52:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:52:52 DEBUG }, 08:52:52 DEBUG "value": [ 08:52:52 DEBUG 08:52:52 DEBUG 08:52:52 DEBUG 08:52:52 DEBUG [...] 08:52:52 DEBUG 08:52:52 DEBUG 08:52:52 DEBUG 08:52:52 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:52:52 DEBUG }, 08:52:52 DEBUG "value": [ 08:52:52 DEBUG 1684569172, 08:52:52 DEBUG "1.0330146248923742" 08:52:52 DEBUG ] 08:52:52 DEBUG } 08:52:52 DEBUG ] 08:52:52 DEBUG } 08:52:52 DEBUG } 127.0.0.1 - - [20/May/2023 08:52:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:53:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:53:08] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:53:14] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:53:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:53:28] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:53:28 INFO 08:53:28 INFO [loop_until]: kubectl --namespace=xlou top pods 08:53:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:53:28 INFO [loop_until]: OK (rc = 0) 08:53:28 DEBUG --- stdout --- 08:53:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 368m 9408Mi am-7849cf7bdb-lxfkl 313m 9342Mi am-7849cf7bdb-td7js 312m 9402Mi ds-cts-0 1242m 13283Mi ds-cts-1 675m 13402Mi ds-cts-2 740m 13527Mi ds-idrepo-0 237m 13662Mi ds-idrepo-1 13m 13554Mi ds-idrepo-2 14m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 36m 77Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 986m 4991Mi 08:53:28 DEBUG --- stderr --- 08:53:28 DEBUG 127.0.0.1 - - [20/May/2023 08:53:34] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:53:40 INFO 08:53:40 INFO [loop_until]: kubectl --namespace=xlou top node 08:53:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:53:40 INFO [loop_until]: OK (rc = 0) 08:53:40 DEBUG --- stdout --- 08:53:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 392m 2% 11556Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 75m 0% 4950Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 468m 2% 11674Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 397m 2% 12009Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 163m 1% 3089Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 122m 0% 1371Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5298Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14646Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 913m 5% 14798Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 287m 1% 14775Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 69m 0% 14582Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 855m 5% 14884Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 994m 6% 14648Mi 24% gke-xlou-cdm-frontend-f357e02a-1hq2 1077m 6% 7118Mi 12% 08:53:40 DEBUG --- stderr --- 08:53:40 DEBUG 127.0.0.1 - - [20/May/2023 08:53:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:53:48] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "ds-idrepo-2" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "ds-idrepo-1" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "ds-cts-2" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "ds-idrepo-1" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "ds-idrepo-2" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "ds-idrepo-1" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "ds-cts-2" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "ds-idrepo-1" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "ds-idrepo-2" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "ds-idrepo-1" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569231" 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0.2" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG ] 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG } 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG } 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG { 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "data": { 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG "result": [ 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG { 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:53:51 DEBUG "pod": "ds-cts-2" 08:53:51 DEBUG "instance": "10.142.0.17:9100" 08:53:51 DEBUG }, 08:53:51 DEBUG }, 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG "value": [ 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG [...] 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:53:51 DEBUG "pod": "ds-idrepo-1" 08:53:51 DEBUG "instance": "10.142.0.61:9100" 08:53:51 DEBUG }, 08:53:51 DEBUG }, 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569231" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG "value": [ 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG "0" 08:53:51 DEBUG "6656.533333333334" 08:53:51 DEBUG ] 08:53:51 DEBUG ] 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG ] 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "instance": "10.142.0.17:9100" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "instance": "10.142.0.61:9100" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0.00040000000000001514" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "instance": "10.142.0.17:9100" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "instance": "10.142.0.61:9100" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569231" 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "14259687424" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "instance": "10.142.0.17:9100" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "instance": "10.142.0.61:9100" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "2863.9" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:51 INFO [http_cmd]: http status code OK 08:53:51 DEBUG --- status code --- 08:53:51 DEBUG http status code is 200 (expected 200) 08:53:51 DEBUG --- http response --- 08:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:53:51 DEBUG ----- output ----- 08:53:51 DEBUG { 08:53:51 DEBUG "status": "success", 08:53:51 DEBUG "data": { 08:53:51 DEBUG "resultType": "vector", 08:53:51 DEBUG "result": [ 08:53:51 DEBUG { 08:53:51 DEBUG "metric": { 08:53:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG [...] 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG 08:53:51 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:53:51 DEBUG }, 08:53:51 DEBUG "value": [ 08:53:51 DEBUG 1684569231, 08:53:51 DEBUG "0.0046149588836946675" 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG ] 08:53:51 DEBUG } 08:53:51 DEBUG } 08:53:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569231" 08:53:52 INFO [http_cmd]: http status code OK 08:53:52 DEBUG --- status code --- 08:53:52 DEBUG http status code is 200 (expected 200) 08:53:52 DEBUG --- http response --- 08:53:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:53:52 DEBUG ----- output ----- 08:53:52 DEBUG { 08:53:52 DEBUG "status": "success", 08:53:52 DEBUG "data": { 08:53:52 DEBUG "resultType": "vector", 08:53:52 DEBUG "result": [ 08:53:52 DEBUG { 08:53:52 DEBUG "metric": { 08:53:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:53:52 DEBUG }, 08:53:52 DEBUG "value": [ 08:53:52 DEBUG 08:53:52 DEBUG 08:53:52 DEBUG 08:53:52 DEBUG [...] 08:53:52 DEBUG 08:53:52 DEBUG 08:53:52 DEBUG 08:53:52 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:53:52 DEBUG }, 08:53:52 DEBUG "value": [ 08:53:52 DEBUG 1684569231, 08:53:52 DEBUG "0" 08:53:52 DEBUG ] 08:53:52 DEBUG } 08:53:52 DEBUG ] 08:53:52 DEBUG } 08:53:52 DEBUG } 08:53:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569232" 08:53:52 INFO [http_cmd]: http status code OK 08:53:52 DEBUG --- status code --- 08:53:52 DEBUG http status code is 200 (expected 200) 08:53:52 DEBUG --- http response --- 08:53:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:53:52 DEBUG ----- output ----- 08:53:52 DEBUG { 08:53:52 DEBUG "status": "success", 08:53:52 DEBUG "data": { 08:53:52 DEBUG "resultType": "vector", 08:53:52 DEBUG "result": [ 08:53:52 DEBUG { 08:53:52 DEBUG "metric": { 08:53:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:53:52 DEBUG }, 08:53:52 DEBUG "value": [ 08:53:52 DEBUG 08:53:52 DEBUG 08:53:52 DEBUG 08:53:52 DEBUG [...] 08:53:52 DEBUG 08:53:52 DEBUG 08:53:52 DEBUG 08:53:52 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:53:52 DEBUG }, 08:53:52 DEBUG "value": [ 08:53:52 DEBUG 1684569232, 08:53:52 DEBUG "1.0354557886149043" 08:53:52 DEBUG ] 08:53:52 DEBUG } 08:53:52 DEBUG ] 08:53:52 DEBUG } 08:53:52 DEBUG } 127.0.0.1 - - [20/May/2023 08:53:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:54:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:54:07] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:54:14] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:54:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:54:28] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:54:29 INFO 08:54:29 INFO [loop_until]: kubectl --namespace=xlou top pods 08:54:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:54:29 INFO [loop_until]: OK (rc = 0) 08:54:29 DEBUG --- stdout --- 08:54:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 360m 9408Mi am-7849cf7bdb-lxfkl 291m 9342Mi am-7849cf7bdb-td7js 307m 9402Mi ds-cts-0 909m 13301Mi ds-cts-1 684m 13414Mi ds-cts-2 637m 13525Mi ds-idrepo-0 233m 13662Mi ds-idrepo-1 12m 13554Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 44m 76Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 989m 4991Mi 08:54:29 DEBUG --- stderr --- 08:54:29 DEBUG 127.0.0.1 - - [20/May/2023 08:54:34] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:54:40 INFO 08:54:40 INFO [loop_until]: kubectl --namespace=xlou top node 08:54:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:54:41 INFO [loop_until]: OK (rc = 0) 08:54:41 DEBUG --- stdout --- 08:54:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 372m 2% 11559Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 77m 0% 4953Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 452m 2% 11677Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 399m 2% 12008Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 158m 0% 3129Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 121m 0% 1369Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 88m 0% 5300Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14651Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 1319m 8% 14783Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 252m 1% 14778Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 70m 0% 14582Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 919m 5% 14922Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 680m 4% 14665Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1077m 6% 7120Mi 12% 08:54:41 DEBUG --- stderr --- 08:54:41 DEBUG 127.0.0.1 - - [20/May/2023 08:54:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:54:47] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "ds-idrepo-2" 08:54:51 DEBUG "pod": "ds-cts-2" 08:54:51 DEBUG }, 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "ds-idrepo-1" 08:54:51 DEBUG "pod": "ds-idrepo-1" 08:54:51 DEBUG }, 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "ds-idrepo-1" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "10815.949847335247" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "ds-idrepo-2" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "ds-idrepo-1" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "ds-idrepo-1" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "ds-cts-2" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "ds-idrepo-1" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "instance": "10.142.0.17:9100" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "instance": "10.142.0.61:9100" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "6635.833333333333" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569291" 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "ds-cts-2" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "ds-idrepo-1" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "ds-idrepo-2" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "ds-idrepo-1" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "14295965696" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0.2" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "instance": "10.142.0.17:9100" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "instance": "10.142.0.61:9100" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0.00036666666666747006" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "instance": "10.142.0.17:9100" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "instance": "10.142.0.61:9100" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569291" 08:54:51 INFO [http_cmd]: http status code OK 08:54:51 DEBUG --- status code --- 08:54:51 DEBUG http status code is 200 (expected 200) 08:54:51 DEBUG --- http response --- 08:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:54:51 DEBUG ----- output ----- 08:54:51 DEBUG { 08:54:51 DEBUG "status": "success", 08:54:51 DEBUG "data": { 08:54:51 DEBUG "resultType": "vector", 08:54:51 DEBUG "result": [ 08:54:51 DEBUG { 08:54:51 DEBUG "metric": { 08:54:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG [...] 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG 08:54:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:54:51 DEBUG }, 08:54:51 DEBUG "value": [ 08:54:51 DEBUG 1684569291, 08:54:51 DEBUG "0" 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG ] 08:54:51 DEBUG } 08:54:51 DEBUG } 08:54:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569291" 08:54:52 INFO [http_cmd]: http status code OK 08:54:52 DEBUG --- status code --- 08:54:52 DEBUG http status code is 200 (expected 200) 08:54:52 DEBUG --- http response --- 08:54:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:54:52 DEBUG ----- output ----- 08:54:52 DEBUG { 08:54:52 DEBUG "status": "success", 08:54:52 DEBUG "data": { 08:54:52 DEBUG "resultType": "vector", 08:54:52 DEBUG "result": [ 08:54:52 DEBUG { 08:54:52 DEBUG "metric": { 08:54:52 DEBUG "instance": "10.142.0.17:9100" 08:54:52 DEBUG }, 08:54:52 DEBUG "value": [ 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG [...] 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG "instance": "10.142.0.61:9100" 08:54:52 DEBUG }, 08:54:52 DEBUG "value": [ 08:54:52 DEBUG 1684569291, 08:54:52 DEBUG "2949.3" 08:54:52 DEBUG ] 08:54:52 DEBUG } 08:54:52 DEBUG ] 08:54:52 DEBUG } 08:54:52 DEBUG } 08:54:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569292" 08:54:52 INFO [http_cmd]: http status code OK 08:54:52 DEBUG --- status code --- 08:54:52 DEBUG http status code is 200 (expected 200) 08:54:52 DEBUG --- http response --- 08:54:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:54:52 DEBUG ----- output ----- 08:54:52 DEBUG { 08:54:52 DEBUG "status": "success", 08:54:52 DEBUG "data": { 08:54:52 DEBUG "resultType": "vector", 08:54:52 DEBUG "result": [ 08:54:52 DEBUG { 08:54:52 DEBUG "metric": { 08:54:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:54:52 DEBUG }, 08:54:52 DEBUG "value": [ 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG [...] 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG "pod": "ds-idrepo-0" 08:54:52 DEBUG }, 08:54:52 DEBUG "value": [ 08:54:52 DEBUG 1684569292, 08:54:52 DEBUG "0" 08:54:52 DEBUG ] 08:54:52 DEBUG } 08:54:52 DEBUG ] 08:54:52 DEBUG } 08:54:52 DEBUG } 08:54:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569292" 08:54:52 INFO [http_cmd]: http status code OK 08:54:52 DEBUG --- status code --- 08:54:52 DEBUG http status code is 200 (expected 200) 08:54:52 DEBUG --- http response --- 08:54:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:54:52 DEBUG ----- output ----- 08:54:52 DEBUG { 08:54:52 DEBUG "status": "success", 08:54:52 DEBUG "data": { 08:54:52 DEBUG "resultType": "vector", 08:54:52 DEBUG "result": [ 08:54:52 DEBUG { 08:54:52 DEBUG "metric": { 08:54:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:54:52 DEBUG }, 08:54:52 DEBUG "value": [ 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG [...] 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG "pod": "idm-78f4b47cb9-7z7ld" 08:54:52 DEBUG }, 08:54:52 DEBUG "value": [ 08:54:52 DEBUG 1684569292, 08:54:52 DEBUG "0" 08:54:52 DEBUG ] 08:54:52 DEBUG } 08:54:52 DEBUG ] 08:54:52 DEBUG } 08:54:52 DEBUG } 08:54:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569292" 08:54:52 INFO [http_cmd]: http status code OK 08:54:52 DEBUG --- status code --- 08:54:52 DEBUG http status code is 200 (expected 200) 08:54:52 DEBUG --- http response --- 08:54:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:54:52 DEBUG ----- output ----- 08:54:52 DEBUG { 08:54:52 DEBUG "status": "success", 08:54:52 DEBUG "data": { 08:54:52 DEBUG "resultType": "vector", 08:54:52 DEBUG "result": [ 08:54:52 DEBUG { 08:54:52 DEBUG "metric": { 08:54:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:54:52 DEBUG }, 08:54:52 DEBUG "value": [ 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG [...] 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG 08:54:52 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:54:52 DEBUG }, 08:54:52 DEBUG "value": [ 08:54:52 DEBUG 1684569292, 08:54:52 DEBUG "1.0245859794986139" 08:54:52 DEBUG ] 08:54:52 DEBUG } 08:54:52 DEBUG ] 08:54:52 DEBUG } 08:54:52 DEBUG } 127.0.0.1 - - [20/May/2023 08:54:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:55:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:55:07] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:55:14] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:55:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:55:27] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:55:29 INFO 08:55:29 INFO [loop_until]: kubectl --namespace=xlou top pods 08:55:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:55:29 INFO [loop_until]: OK (rc = 0) 08:55:29 DEBUG --- stdout --- 08:55:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 299m 9408Mi am-7849cf7bdb-lxfkl 234m 9342Mi am-7849cf7bdb-td7js 256m 9402Mi ds-cts-0 624m 13289Mi ds-cts-1 1208m 13390Mi ds-cts-2 1077m 13439Mi ds-idrepo-0 36m 13662Mi ds-idrepo-1 12m 13555Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 43m 76Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1006m 4991Mi 08:55:29 DEBUG --- stderr --- 08:55:29 DEBUG 127.0.0.1 - - [20/May/2023 08:55:34] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:55:41 INFO 08:55:41 INFO [loop_until]: kubectl --namespace=xlou top node 08:55:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 127.0.0.1 - - [20/May/2023 08:55:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:55:41 INFO [loop_until]: OK (rc = 0) 08:55:41 DEBUG --- stdout --- 08:55:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 186m 1% 11557Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 81m 0% 4950Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 224m 1% 11675Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 210m 1% 12010Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 150m 0% 3134Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 122m 0% 1369Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 5300Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14652Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 613m 3% 14750Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 82m 0% 14776Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14582Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 728m 4% 14807Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 958m 6% 14663Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1084m 6% 7119Mi 12% 08:55:41 DEBUG --- stderr --- 08:55:41 DEBUG 127.0.0.1 - - [20/May/2023 08:55:47] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "pod": "ds-idrepo-2" 08:55:51 DEBUG "pod": "ds-cts-2" 08:55:51 DEBUG }, 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "pod": "ds-idrepo-1" 08:55:51 DEBUG "pod": "ds-idrepo-1" 08:55:51 DEBUG }, 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG } 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "pod": "ds-idrepo-1" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "pod": "ds-idrepo-2" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "pod": "ds-idrepo-1" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "pod": "ds-cts-2" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "pod": "ds-idrepo-1" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569351" 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569351" 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG "result": [ 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG { 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG { 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG "metric": { 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG "metric": { 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG "instance": "10.142.0.17:9100" 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG "instance": "10.142.0.17:9100" 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG }, 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG }, 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:55:51 DEBUG "value": [ 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:55:51 DEBUG "value": [ 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG 08:55:51 DEBUG { 08:55:51 DEBUG 08:55:51 DEBUG { 08:55:51 DEBUG 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG 08:55:51 DEBUG "data": { 08:55:51 DEBUG [...] 08:55:51 DEBUG "data": { 08:55:51 DEBUG [...] 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG 08:55:51 DEBUG "result": [ 08:55:51 DEBUG 08:55:51 DEBUG "result": [ 08:55:51 DEBUG 08:55:51 DEBUG { 08:55:51 DEBUG 08:55:51 DEBUG { 08:55:51 DEBUG 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "instance": "10.142.0.61:9100" 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "instance": "10.142.0.61:9100" 08:55:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:55:51 DEBUG }, 08:55:51 DEBUG "pod": "ds-idrepo-2" 08:55:51 DEBUG }, 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG 08:55:51 DEBUG "0.0000999999999995301" 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG 08:55:51 DEBUG "6815.5" 08:55:51 DEBUG 08:55:51 DEBUG ] 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG 08:55:51 DEBUG ] 08:55:51 DEBUG 08:55:51 DEBUG } 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:55:51 DEBUG 08:55:51 DEBUG } 08:55:51 DEBUG [...] 08:55:51 DEBUG ] 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG [...] 08:55:51 DEBUG ] 08:55:51 DEBUG 08:55:51 DEBUG } 08:55:51 DEBUG { 08:55:51 DEBUG 08:55:51 DEBUG } 08:55:51 DEBUG 08:55:51 DEBUG } 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG 08:55:51 DEBUG } 08:55:51 DEBUG 08:55:51 DEBUG "data": { 08:55:51 DEBUG 08:55:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "pod": "ds-idrepo-1" 08:55:51 DEBUG }, 08:55:51 DEBUG "result": [ 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG { 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "metric": { 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0.2" 08:55:51 DEBUG "pod": "ds-cts-2" 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG }, 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG "value": [ 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 DEBUG 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569351" 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "pod": "ds-idrepo-1" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "14180380672" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569351" 08:55:51 INFO [http_cmd]: http status code OK 08:55:51 DEBUG --- status code --- 08:55:51 DEBUG http status code is 200 (expected 200) 08:55:51 DEBUG --- http response --- 08:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:55:51 DEBUG ----- output ----- 08:55:51 DEBUG { 08:55:51 DEBUG "status": "success", 08:55:51 DEBUG "data": { 08:55:51 DEBUG "resultType": "vector", 08:55:51 DEBUG "result": [ 08:55:51 DEBUG { 08:55:51 DEBUG "metric": { 08:55:51 DEBUG "instance": "10.142.0.17:9100" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG [...] 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG 08:55:51 DEBUG "instance": "10.142.0.61:9100" 08:55:51 DEBUG }, 08:55:51 DEBUG "value": [ 08:55:51 DEBUG 1684569351, 08:55:51 DEBUG "0" 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG ] 08:55:51 DEBUG } 08:55:51 DEBUG } 08:55:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569351" 08:55:52 INFO [http_cmd]: http status code OK 08:55:52 DEBUG --- status code --- 08:55:52 DEBUG http status code is 200 (expected 200) 08:55:52 DEBUG --- http response --- 08:55:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:55:52 DEBUG ----- output ----- 08:55:52 DEBUG { 08:55:52 DEBUG "status": "success", 08:55:52 DEBUG "data": { 08:55:52 DEBUG "resultType": "vector", 08:55:52 DEBUG "result": [ 08:55:52 DEBUG { 08:55:52 DEBUG "metric": { 08:55:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG [...] 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 1684569351, 08:55:52 DEBUG "0" 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG } 08:55:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569352" 08:55:52 INFO [http_cmd]: http status code OK 08:55:52 DEBUG --- status code --- 08:55:52 DEBUG http status code is 200 (expected 200) 08:55:52 DEBUG --- http response --- 08:55:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:55:52 DEBUG ----- output ----- 08:55:52 DEBUG { 08:55:52 DEBUG "status": "success", 08:55:52 DEBUG "data": { 08:55:52 DEBUG "resultType": "vector", 08:55:52 DEBUG "result": [ 08:55:52 DEBUG { 08:55:52 DEBUG "metric": { 08:55:52 DEBUG "instance": "10.142.0.17:9100" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG [...] 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG "instance": "10.142.0.61:9100" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 1684569352, 08:55:52 DEBUG "5304.8" 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG } 08:55:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569352" 08:55:52 INFO [http_cmd]: http status code OK 08:55:52 DEBUG --- status code --- 08:55:52 DEBUG http status code is 200 (expected 200) 08:55:52 DEBUG --- http response --- 08:55:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:55:52 DEBUG ----- output ----- 08:55:52 DEBUG { 08:55:52 DEBUG "status": "success", 08:55:52 DEBUG "data": { 08:55:52 DEBUG "resultType": "vector", 08:55:52 DEBUG "result": [ 08:55:52 DEBUG { 08:55:52 DEBUG "metric": { 08:55:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG [...] 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG "pod": "ds-idrepo-1" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 1684569352, 08:55:52 DEBUG "0" 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG } 08:55:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569352" 08:55:52 INFO [http_cmd]: http status code OK 08:55:52 DEBUG --- status code --- 08:55:52 DEBUG http status code is 200 (expected 200) 08:55:52 DEBUG --- http response --- 08:55:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:55:52 DEBUG ----- output ----- 08:55:52 DEBUG { 08:55:52 DEBUG "status": "success", 08:55:52 DEBUG "data": { 08:55:52 DEBUG "resultType": "vector", 08:55:52 DEBUG "result": [ 08:55:52 DEBUG { 08:55:52 DEBUG "metric": { 08:55:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG [...] 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 1684569352, 08:55:52 DEBUG "0" 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG } 08:55:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569352" 08:55:52 INFO [http_cmd]: http status code OK 08:55:52 DEBUG --- status code --- 08:55:52 DEBUG http status code is 200 (expected 200) 08:55:52 DEBUG --- http response --- 08:55:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:55:52 DEBUG ----- output ----- 08:55:52 DEBUG { 08:55:52 DEBUG "status": "success", 08:55:52 DEBUG "data": { 08:55:52 DEBUG "resultType": "vector", 08:55:52 DEBUG "result": [ 08:55:52 DEBUG { 08:55:52 DEBUG "metric": { 08:55:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG [...] 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG 08:55:52 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:55:52 DEBUG }, 08:55:52 DEBUG "value": [ 08:55:52 DEBUG 1684569352, 08:55:52 DEBUG "1.0398038882008602" 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG ] 08:55:52 DEBUG } 08:55:52 DEBUG } 127.0.0.1 - - [20/May/2023 08:55:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:56:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:56:07] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:56:14] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:56:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:56:27] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:56:29 INFO 08:56:29 INFO [loop_until]: kubectl --namespace=xlou top pods 08:56:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:56:29 INFO [loop_until]: OK (rc = 0) 08:56:29 DEBUG --- stdout --- 08:56:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 13m 9407Mi am-7849cf7bdb-lxfkl 11m 9342Mi am-7849cf7bdb-td7js 11m 9402Mi ds-cts-0 65m 13286Mi ds-cts-1 80m 13378Mi ds-cts-2 302m 13475Mi ds-idrepo-0 22m 13662Mi ds-idrepo-1 13m 13555Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 5m 3395Mi idm-78f4b47cb9-fmzjg 8m 3297Mi lodemon-58bb5f9bf4-7mmfr 44m 77Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1008m 4991Mi 08:56:29 DEBUG --- stderr --- 08:56:29 DEBUG 127.0.0.1 - - [20/May/2023 08:56:34] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:56:41 INFO 08:56:41 INFO [loop_until]: kubectl --namespace=xlou top node 127.0.0.1 - - [20/May/2023 08:56:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:56:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:56:41 INFO [loop_until]: OK (rc = 0) 08:56:41 DEBUG --- stdout --- 08:56:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 75m 0% 11558Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 81m 0% 4952Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 97m 0% 11678Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 89m 0% 12010Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 159m 1% 3139Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 124m 0% 1371Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 70m 0% 14650Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 131m 0% 14751Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 71m 0% 14773Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 69m 0% 14584Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 136m 0% 14837Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 137m 0% 14666Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1079m 6% 7124Mi 12% 08:56:41 DEBUG --- stderr --- 08:56:41 DEBUG 127.0.0.1 - - [20/May/2023 08:56:48] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG { 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "data": { 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "result": [ 08:56:51 DEBUG { 08:56:51 DEBUG "metric": { 08:56:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG [...] 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG "0" 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG { 08:56:51 DEBUG { 08:56:51 DEBUG { 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "data": { 08:56:51 DEBUG "data": { 08:56:51 DEBUG "data": { 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "result": [ 08:56:51 DEBUG "result": [ 08:56:51 DEBUG "result": [ 08:56:51 DEBUG { 08:56:51 DEBUG { 08:56:51 DEBUG { 08:56:51 DEBUG "metric": { 08:56:51 DEBUG "metric": { 08:56:51 DEBUG "metric": { 08:56:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:56:51 DEBUG "pod": "ds-idrepo-2" 08:56:51 DEBUG "pod": "ds-cts-2" 08:56:51 DEBUG }, 08:56:51 DEBUG }, 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG "value": [ 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG [...] 08:56:51 DEBUG [...] 08:56:51 DEBUG [...] 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "pod": "ds-idrepo-0" 08:56:51 DEBUG "pod": "ds-idrepo-1" 08:56:51 DEBUG "pod": "ds-idrepo-1" 08:56:51 DEBUG }, 08:56:51 DEBUG }, 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG "value": [ 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG "1876.1847715020613" 08:56:51 DEBUG "0" 08:56:51 DEBUG "0" 08:56:51 DEBUG ] 08:56:51 DEBUG ] 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 DEBUG ] 08:56:51 DEBUG ] 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG { 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "data": { 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "result": [ 08:56:51 DEBUG { 08:56:51 DEBUG "metric": { 08:56:51 DEBUG "pod": "ds-idrepo-2" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG [...] 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "pod": "ds-idrepo-1" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG "0" 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG { 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "data": { 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "result": [ 08:56:51 DEBUG { 08:56:51 DEBUG "metric": { 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG [...] 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "pod": "ds-idrepo-0" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG "0" 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG { 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "data": { 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "result": [ 08:56:51 DEBUG { 08:56:51 DEBUG "metric": { 08:56:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG [...] 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG "0" 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG { 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "data": { 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "result": [ 08:56:51 DEBUG { 08:56:51 DEBUG "metric": { 08:56:51 DEBUG "pod": "ds-cts-2" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG [...] 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "pod": "ds-idrepo-1" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG "0" 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569411" 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG --- http response --- 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG { 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "data": { 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG "result": [ 08:56:51 DEBUG { 08:56:51 DEBUG { 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "metric": { 08:56:51 DEBUG "data": { 08:56:51 DEBUG "pod": "ds-cts-2" 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG }, 08:56:51 DEBUG "result": [ 08:56:51 DEBUG "value": [ 08:56:51 DEBUG { 08:56:51 DEBUG 08:56:51 DEBUG "metric": { 08:56:51 DEBUG 08:56:51 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:56:51 DEBUG 08:56:51 DEBUG }, 08:56:51 DEBUG [...] 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "pod": "ds-idrepo-1" 08:56:51 DEBUG [...] 08:56:51 DEBUG }, 08:56:51 DEBUG 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG 08:56:51 DEBUG "0" 08:56:51 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:56:51 DEBUG ] 08:56:51 DEBUG }, 08:56:51 DEBUG } 08:56:51 DEBUG "value": [ 08:56:51 DEBUG ] 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG } 08:56:51 DEBUG "14212673536" 08:56:51 DEBUG } 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569411" 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569411" 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569411" 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG { 08:56:51 DEBUG { 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG "data": { 08:56:51 DEBUG "data": { 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG "result": [ 08:56:51 DEBUG "result": [ 08:56:51 DEBUG { 08:56:51 DEBUG { 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG "metric": { 08:56:51 DEBUG "metric": { 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG "instance": "10.142.0.17:9100" 08:56:51 DEBUG "instance": "10.142.0.17:9100" 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG }, 08:56:51 DEBUG }, 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG "value": [ 08:56:51 DEBUG "value": [ 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG { 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG [...] 08:56:51 DEBUG [...] 08:56:51 DEBUG "data": { 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "resultType": "vector", 08:56:51 INFO [http_cmd]: http status code OK 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG "result": [ 08:56:51 DEBUG --- status code --- 08:56:51 DEBUG 08:56:51 DEBUG 08:56:51 DEBUG { 08:56:51 DEBUG http status code is 200 (expected 200) 08:56:51 DEBUG "instance": "10.142.0.61:9100" 08:56:51 DEBUG "instance": "10.142.0.61:9100" 08:56:51 DEBUG "metric": { 08:56:51 DEBUG --- http response --- 08:56:51 DEBUG }, 08:56:51 DEBUG }, 08:56:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:56:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:56:51 DEBUG "value": [ 08:56:51 DEBUG "value": [ 08:56:51 DEBUG }, 08:56:51 DEBUG ----- output ----- 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG { 08:56:51 DEBUG "6645.5" 08:56:51 DEBUG "0.0004999999999995453" 08:56:51 DEBUG 08:56:51 DEBUG "status": "success", 08:56:51 DEBUG ] 08:56:51 DEBUG ] 08:56:51 DEBUG 08:56:51 DEBUG "data": { 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 DEBUG 08:56:51 DEBUG "resultType": "vector", 08:56:51 DEBUG ] 08:56:51 DEBUG ] 08:56:51 DEBUG [...] 08:56:51 DEBUG "result": [ 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 DEBUG 08:56:51 DEBUG { 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:51 DEBUG 08:56:51 DEBUG "metric": { 08:56:51 DEBUG 08:56:51 DEBUG "pod": "ds-idrepo-2" 08:56:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:56:51 DEBUG }, 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG 08:56:51 DEBUG "0.2" 08:56:51 DEBUG 08:56:51 DEBUG ] 08:56:51 DEBUG [...] 08:56:51 DEBUG } 08:56:51 DEBUG 08:56:51 DEBUG ] 08:56:51 DEBUG 08:56:51 DEBUG } 08:56:51 DEBUG 08:56:51 DEBUG } 08:56:51 DEBUG "pod": "ds-idrepo-1" 08:56:51 DEBUG }, 08:56:51 DEBUG "value": [ 08:56:51 DEBUG 1684569411, 08:56:51 DEBUG "0" 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG ] 08:56:51 DEBUG } 08:56:51 DEBUG } 08:56:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569412" 08:56:52 INFO [http_cmd]: http status code OK 08:56:52 DEBUG --- status code --- 08:56:52 DEBUG http status code is 200 (expected 200) 08:56:52 DEBUG --- http response --- 08:56:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:56:52 DEBUG ----- output ----- 08:56:52 DEBUG { 08:56:52 DEBUG "status": "success", 08:56:52 DEBUG "data": { 08:56:52 DEBUG "resultType": "vector", 08:56:52 DEBUG "result": [ 08:56:52 DEBUG { 08:56:52 DEBUG "metric": { 08:56:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG [...] 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 1684569412, 08:56:52 DEBUG "0" 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG } 08:56:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569412" 08:56:52 INFO [http_cmd]: http status code OK 08:56:52 DEBUG --- status code --- 08:56:52 DEBUG http status code is 200 (expected 200) 08:56:52 DEBUG --- http response --- 08:56:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:56:52 DEBUG ----- output ----- 08:56:52 DEBUG { 08:56:52 DEBUG "status": "success", 08:56:52 DEBUG "data": { 08:56:52 DEBUG "resultType": "vector", 08:56:52 DEBUG "result": [ 08:56:52 DEBUG { 08:56:52 DEBUG "metric": { 08:56:52 DEBUG "instance": "10.142.0.17:9100" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG [...] 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG "instance": "10.142.0.61:9100" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 1684569412, 08:56:52 DEBUG "0" 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG } 08:56:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569412" 08:56:52 INFO [http_cmd]: http status code OK 08:56:52 DEBUG --- status code --- 08:56:52 DEBUG http status code is 200 (expected 200) 08:56:52 DEBUG --- http response --- 08:56:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:56:52 DEBUG ----- output ----- 08:56:52 DEBUG { 08:56:52 DEBUG "status": "success", 08:56:52 DEBUG "data": { 08:56:52 DEBUG "resultType": "vector", 08:56:52 DEBUG "result": [ 08:56:52 DEBUG { 08:56:52 DEBUG "metric": { 08:56:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG [...] 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 1684569412, 08:56:52 DEBUG "0" 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG } 08:56:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569412" 08:56:52 INFO [http_cmd]: http status code OK 08:56:52 DEBUG --- status code --- 08:56:52 DEBUG http status code is 200 (expected 200) 08:56:52 DEBUG --- http response --- 08:56:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:56:52 DEBUG ----- output ----- 08:56:52 DEBUG { 08:56:52 DEBUG "status": "success", 08:56:52 DEBUG "data": { 08:56:52 DEBUG "resultType": "vector", 08:56:52 DEBUG "result": [ 08:56:52 DEBUG { 08:56:52 DEBUG "metric": { 08:56:52 DEBUG "instance": "10.142.0.17:9100" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG [...] 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG "instance": "10.142.0.61:9100" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 1684569412, 08:56:52 DEBUG "2963.133333333333" 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG } 08:56:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569412" 08:56:52 INFO [http_cmd]: http status code OK 08:56:52 DEBUG --- status code --- 08:56:52 DEBUG http status code is 200 (expected 200) 08:56:52 DEBUG --- http response --- 08:56:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:56:52 DEBUG ----- output ----- 08:56:52 DEBUG { 08:56:52 DEBUG "status": "success", 08:56:52 DEBUG "data": { 08:56:52 DEBUG "resultType": "vector", 08:56:52 DEBUG "result": [ 08:56:52 DEBUG { 08:56:52 DEBUG "metric": { 08:56:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG [...] 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG "pod": "ds-idrepo-0" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 1684569412, 08:56:52 DEBUG "0.015024493816083237" 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG } 08:56:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569412" 08:56:52 INFO [http_cmd]: http status code OK 08:56:52 DEBUG --- status code --- 08:56:52 DEBUG http status code is 200 (expected 200) 08:56:52 DEBUG --- http response --- 08:56:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 08:56:52 DEBUG ----- output ----- 08:56:52 DEBUG { 08:56:52 DEBUG "status": "success", 08:56:52 DEBUG "data": { 08:56:52 DEBUG "resultType": "vector", 08:56:52 DEBUG "result": [ 08:56:52 DEBUG { 08:56:52 DEBUG "metric": { 08:56:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG [...] 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG "pod": "ds-idrepo-0" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 1684569412, 08:56:52 DEBUG "0.4580529227299954" 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG } 08:56:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569412" 08:56:52 INFO [http_cmd]: http status code OK 08:56:52 DEBUG --- status code --- 08:56:52 DEBUG http status code is 200 (expected 200) 08:56:52 DEBUG --- http response --- 08:56:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:56:52 DEBUG ----- output ----- 08:56:52 DEBUG { 08:56:52 DEBUG "status": "success", 08:56:52 DEBUG "data": { 08:56:52 DEBUG "resultType": "vector", 08:56:52 DEBUG "result": [ 08:56:52 DEBUG { 08:56:52 DEBUG "metric": { 08:56:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG [...] 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG 08:56:52 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:56:52 DEBUG }, 08:56:52 DEBUG "value": [ 08:56:52 DEBUG 1684569412, 08:56:52 DEBUG "1.0298576350884747" 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG ] 08:56:52 DEBUG } 08:56:52 DEBUG } 127.0.0.1 - - [20/May/2023 08:56:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:57:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:57:08] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:57:14] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:57:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:57:27] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:57:29 INFO 08:57:29 INFO [loop_until]: kubectl --namespace=xlou top pods 08:57:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:57:29 INFO [loop_until]: OK (rc = 0) 08:57:29 DEBUG --- stdout --- 08:57:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 12m 9404Mi am-7849cf7bdb-lxfkl 11m 9340Mi am-7849cf7bdb-td7js 11m 9400Mi ds-cts-0 88m 13289Mi ds-cts-1 77m 13379Mi ds-cts-2 80m 13479Mi ds-idrepo-0 17m 13662Mi ds-idrepo-1 12m 13555Mi ds-idrepo-2 19m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 7m 3395Mi idm-78f4b47cb9-fmzjg 5m 3298Mi lodemon-58bb5f9bf4-7mmfr 49m 77Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 997m 4991Mi 08:57:29 DEBUG --- stderr --- 08:57:29 DEBUG 127.0.0.1 - - [20/May/2023 08:57:34] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:57:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:57:41 INFO 08:57:41 INFO [loop_until]: kubectl --namespace=xlou top node 08:57:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:57:41 INFO [loop_until]: OK (rc = 0) 08:57:41 DEBUG --- stdout --- 08:57:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 72m 0% 11556Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 93m 0% 4952Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 99m 0% 11664Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 87m 0% 12011Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 166m 1% 3110Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 122m 0% 1373Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 90m 0% 5296Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 63m 0% 14649Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 120m 0% 14755Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 71m 0% 14774Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 69m 0% 14582Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 121m 0% 14842Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 126m 0% 14666Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1079m 6% 7118Mi 12% 08:57:41 DEBUG --- stderr --- 08:57:41 DEBUG 127.0.0.1 - - [20/May/2023 08:57:47] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:57:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569471" 08:57:51 INFO [http_cmd]: http status code OK 08:57:51 DEBUG --- status code --- 08:57:51 DEBUG http status code is 200 (expected 200) 08:57:51 DEBUG --- http response --- 08:57:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:57:51 DEBUG ----- output ----- 08:57:51 DEBUG { 08:57:51 DEBUG "status": "success", 08:57:51 DEBUG "data": { 08:57:51 DEBUG "resultType": "vector", 08:57:51 DEBUG "result": [ 08:57:51 DEBUG { 08:57:51 DEBUG "metric": { 08:57:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG [...] 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 1684569471, 08:57:51 DEBUG "0" 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG } 08:57:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569471" 08:57:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569471" 08:57:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569471" 08:57:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569471" 08:57:51 INFO [http_cmd]: http status code OK 08:57:51 DEBUG --- status code --- 08:57:51 DEBUG http status code is 200 (expected 200) 08:57:51 DEBUG --- http response --- 08:57:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:57:51 DEBUG ----- output ----- 08:57:51 DEBUG { 08:57:51 DEBUG "status": "success", 08:57:51 DEBUG "data": { 08:57:51 DEBUG "resultType": "vector", 08:57:51 DEBUG "result": [ 08:57:51 DEBUG { 08:57:51 DEBUG "metric": { 08:57:51 DEBUG "pod": "ds-cts-2" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG [...] 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG "pod": "ds-idrepo-1" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 1684569471, 08:57:51 DEBUG "0" 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG } 08:57:51 INFO [http_cmd]: http status code OK 08:57:51 DEBUG --- status code --- 08:57:51 DEBUG http status code is 200 (expected 200) 08:57:51 DEBUG --- http response --- 08:57:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:57:51 DEBUG ----- output ----- 08:57:51 DEBUG { 08:57:51 DEBUG "status": "success", 08:57:51 DEBUG "data": { 08:57:51 DEBUG "resultType": "vector", 08:57:51 DEBUG "result": [ 08:57:51 DEBUG { 08:57:51 DEBUG "metric": { 08:57:51 DEBUG "pod": "ds-idrepo-2" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG [...] 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG "pod": "ds-idrepo-1" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 1684569471, 08:57:51 DEBUG "0" 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG } 08:57:51 INFO [http_cmd]: http status code OK 08:57:51 DEBUG --- status code --- 08:57:51 DEBUG http status code is 200 (expected 200) 08:57:51 DEBUG --- http response --- 08:57:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:57:51 DEBUG ----- output ----- 08:57:51 DEBUG { 08:57:51 DEBUG "status": "success", 08:57:51 DEBUG "data": { 08:57:51 DEBUG "resultType": "vector", 08:57:51 DEBUG "result": [ 08:57:51 DEBUG { 08:57:51 DEBUG "metric": { 08:57:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG [...] 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG "pod": "ds-idrepo-1" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 1684569471, 08:57:51 DEBUG "0" 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG ] 08:57:51 INFO [http_cmd]: http status code OK 08:57:51 DEBUG } 08:57:51 DEBUG --- status code --- 08:57:51 DEBUG } 08:57:51 DEBUG http status code is 200 (expected 200) 08:57:51 DEBUG --- http response --- 08:57:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:57:51 DEBUG ----- output ----- 08:57:51 DEBUG { 08:57:51 DEBUG "status": "success", 08:57:51 DEBUG "data": { 08:57:51 DEBUG "resultType": "vector", 08:57:51 DEBUG "result": [ 08:57:51 DEBUG { 08:57:51 DEBUG "metric": { 08:57:51 DEBUG "pod": "ds-idrepo-2" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG [...] 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG "pod": "ds-idrepo-1" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 1684569471, 08:57:51 DEBUG "0" 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG } 08:57:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569471" 08:57:51 INFO [http_cmd]: http status code OK 08:57:51 DEBUG --- status code --- 08:57:51 DEBUG http status code is 200 (expected 200) 08:57:51 DEBUG --- http response --- 08:57:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:57:51 DEBUG ----- output ----- 08:57:51 DEBUG { 08:57:51 DEBUG "status": "success", 08:57:51 DEBUG "data": { 08:57:51 DEBUG "resultType": "vector", 08:57:51 DEBUG "result": [ 08:57:51 DEBUG { 08:57:51 DEBUG "metric": { 08:57:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG [...] 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 1684569471, 08:57:51 DEBUG "0" 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG } 08:57:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569471" 08:57:51 INFO [http_cmd]: http status code OK 08:57:51 DEBUG --- status code --- 08:57:51 DEBUG http status code is 200 (expected 200) 08:57:51 DEBUG --- http response --- 08:57:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:57:51 DEBUG ----- output ----- 08:57:51 DEBUG { 08:57:51 DEBUG "status": "success", 08:57:51 DEBUG "data": { 08:57:51 DEBUG "resultType": "vector", 08:57:51 DEBUG "result": [ 08:57:51 DEBUG { 08:57:51 DEBUG "metric": { 08:57:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG [...] 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 1684569471, 08:57:51 DEBUG "0" 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG } 08:57:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569471" 08:57:51 INFO [http_cmd]: http status code OK 08:57:51 DEBUG --- status code --- 08:57:51 DEBUG http status code is 200 (expected 200) 08:57:51 DEBUG --- http response --- 08:57:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:57:51 DEBUG ----- output ----- 08:57:51 DEBUG { 08:57:51 DEBUG "status": "success", 08:57:51 DEBUG "data": { 08:57:51 DEBUG "resultType": "vector", 08:57:51 DEBUG "result": [ 08:57:51 DEBUG { 08:57:51 DEBUG "metric": { 08:57:51 DEBUG "pod": "ds-cts-2" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG [...] 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG "pod": "ds-idrepo-1" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 1684569471, 08:57:51 DEBUG "0" 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG } 08:57:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569471" 08:57:51 INFO [http_cmd]: http status code OK 08:57:51 DEBUG --- status code --- 08:57:51 DEBUG http status code is 200 (expected 200) 08:57:51 DEBUG --- http response --- 08:57:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:57:51 DEBUG ----- output ----- 08:57:51 DEBUG { 08:57:51 DEBUG "status": "success", 08:57:51 DEBUG "data": { 08:57:51 DEBUG "resultType": "vector", 08:57:51 DEBUG "result": [ 08:57:51 DEBUG { 08:57:51 DEBUG "metric": { 08:57:51 DEBUG "pod": "ds-cts-2" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG [...] 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG 08:57:51 DEBUG "pod": "ds-idrepo-1" 08:57:51 DEBUG }, 08:57:51 DEBUG "value": [ 08:57:51 DEBUG 1684569471, 08:57:51 DEBUG "0" 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG ] 08:57:51 DEBUG } 08:57:51 DEBUG } 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569472" 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569472" 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG { 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "data": { 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG "result": [ 08:57:52 DEBUG { 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG "14217146368" 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG { 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "data": { 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG "result": [ 08:57:52 DEBUG { 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG "0.2" 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569472" 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569472" 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569472" 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569472" 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569472" 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG { 08:57:52 DEBUG { 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "data": { 08:57:52 DEBUG "data": { 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG "result": [ 08:57:52 DEBUG "result": [ 08:57:52 DEBUG { 08:57:52 DEBUG { 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:57:52 DEBUG "pod": "ds-idrepo-2" 08:57:52 DEBUG }, 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:57:52 DEBUG "pod": "ds-idrepo-1" 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG }, 08:57:52 DEBUG }, 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG "value": [ 08:57:52 DEBUG "value": [ 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG "0" 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG "0" 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:57:52 DEBUG ] 08:57:52 DEBUG { 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG ] 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG } 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG } 08:57:52 DEBUG { 08:57:52 DEBUG ] 08:57:52 DEBUG "data": { 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:57:52 DEBUG ] 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG } 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG } 08:57:52 DEBUG "data": { 08:57:52 DEBUG } 08:57:52 DEBUG "result": [ 08:57:52 DEBUG { 08:57:52 DEBUG } 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG { 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "result": [ 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "data": { 08:57:52 DEBUG { 08:57:52 DEBUG "instance": "10.142.0.17:9100" 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG "metric": { 08:57:52 DEBUG }, 08:57:52 DEBUG "result": [ 08:57:52 DEBUG "instance": "10.142.0.17:9100" 08:57:52 DEBUG "value": [ 08:57:52 DEBUG { 08:57:52 DEBUG }, 08:57:52 DEBUG 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG "instance": "10.142.0.17:9100" 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG }, 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG "instance": "10.142.0.61:9100" 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG }, 08:57:52 DEBUG 08:57:52 DEBUG "instance": "10.142.0.61:9100" 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG }, 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG 08:57:52 DEBUG "value": [ 08:57:52 DEBUG "0" 08:57:52 DEBUG "instance": "10.142.0.61:9100" 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG ] 08:57:52 DEBUG }, 08:57:52 DEBUG "0.00046666666666700016" 08:57:52 DEBUG } 08:57:52 DEBUG "value": [ 08:57:52 DEBUG ] 08:57:52 DEBUG ] 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 DEBUG "6812" 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569472" 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG { 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "data": { 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG "result": [ 08:57:52 DEBUG { 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG "0" 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569472" 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG { 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "data": { 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG "result": [ 08:57:52 DEBUG { 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "instance": "10.142.0.17:9100" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG "instance": "10.142.0.61:9100" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG "5013.900000000001" 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569472" 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG { 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "data": { 08:57:52 DEBUG "resultType": "vector", 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569472" 08:57:52 DEBUG "result": [ 08:57:52 DEBUG { 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG "pod": "overseer-0-6967fc998-8q7zx" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG "0" 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG { 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "data": { 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG "result": [ 08:57:52 DEBUG { 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG "pod": "ds-idrepo-1" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG "0" 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG } 08:57:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569472" 08:57:52 INFO [http_cmd]: http status code OK 08:57:52 DEBUG --- status code --- 08:57:52 DEBUG http status code is 200 (expected 200) 08:57:52 DEBUG --- http response --- 08:57:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:57:52 DEBUG ----- output ----- 08:57:52 DEBUG { 08:57:52 DEBUG "status": "success", 08:57:52 DEBUG "data": { 08:57:52 DEBUG "resultType": "vector", 08:57:52 DEBUG "result": [ 08:57:52 DEBUG { 08:57:52 DEBUG "metric": { 08:57:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG [...] 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG 08:57:52 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:57:52 DEBUG }, 08:57:52 DEBUG "value": [ 08:57:52 DEBUG 1684569472, 08:57:52 DEBUG "1.031835758371143" 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG ] 08:57:52 DEBUG } 08:57:52 DEBUG } 127.0.0.1 - - [20/May/2023 08:57:54] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:58:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:58:08] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:58:15] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:58:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:58:28] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:58:29 INFO 08:58:29 INFO [loop_until]: kubectl --namespace=xlou top pods 08:58:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:58:29 INFO [loop_until]: OK (rc = 0) 08:58:29 DEBUG --- stdout --- 08:58:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 12m 9404Mi am-7849cf7bdb-lxfkl 12m 9340Mi am-7849cf7bdb-td7js 13m 9400Mi ds-cts-0 77m 13290Mi ds-cts-1 75m 13382Mi ds-cts-2 76m 13483Mi ds-idrepo-0 17m 13662Mi ds-idrepo-1 12m 13555Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 4m 3297Mi lodemon-58bb5f9bf4-7mmfr 46m 77Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1002m 4991Mi 08:58:29 DEBUG --- stderr --- 08:58:29 DEBUG 127.0.0.1 - - [20/May/2023 08:58:35] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:58:41 INFO 08:58:41 INFO [loop_until]: kubectl --namespace=xlou top node 08:58:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:58:41 INFO [loop_until]: OK (rc = 0) 08:58:41 DEBUG --- stdout --- 08:58:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 74m 0% 11555Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 4965Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 96m 0% 11679Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 85m 0% 12010Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 150m 0% 3141Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 125m 0% 1372Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5296Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 73m 0% 14652Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 390m 2% 14779Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 75m 0% 14776Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 72m 0% 14581Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 138m 0% 14848Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 134m 0% 14668Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1048m 6% 7118Mi 12% 08:58:41 DEBUG --- stderr --- 08:58:41 DEBUG 127.0.0.1 - - [20/May/2023 08:58:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:58:48] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:58:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569531" 08:58:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569531" 08:58:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569531" 08:58:51 INFO [http_cmd]: http status code OK 08:58:51 DEBUG --- status code --- 08:58:51 DEBUG http status code is 200 (expected 200) 08:58:51 DEBUG --- http response --- 08:58:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:58:51 DEBUG ----- output ----- 08:58:51 DEBUG { 08:58:51 DEBUG "status": "success", 08:58:51 DEBUG "data": { 08:58:51 DEBUG "resultType": "vector", 08:58:51 DEBUG "result": [ 08:58:51 DEBUG { 08:58:51 DEBUG "metric": { 08:58:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG [...] 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 1684569531, 08:58:51 DEBUG "0" 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG } 08:58:51 INFO [http_cmd]: http status code OK 08:58:51 DEBUG --- status code --- 08:58:51 DEBUG http status code is 200 (expected 200) 08:58:51 DEBUG --- http response --- 08:58:51 INFO [http_cmd]: http status code OK 08:58:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:58:51 DEBUG --- status code --- 08:58:51 DEBUG ----- output ----- 08:58:51 DEBUG http status code is 200 (expected 200) 08:58:51 DEBUG { 08:58:51 DEBUG --- http response --- 08:58:51 DEBUG "status": "success", 08:58:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:51 DEBUG "data": { 08:58:51 DEBUG ----- output ----- 08:58:51 DEBUG "resultType": "vector", 08:58:51 DEBUG { 08:58:51 DEBUG "result": [ 08:58:51 DEBUG "status": "success", 08:58:51 DEBUG { 08:58:51 DEBUG "data": { 08:58:51 DEBUG "metric": { 08:58:51 DEBUG "resultType": "vector", 08:58:51 DEBUG "pod": "ds-cts-2" 08:58:51 DEBUG "result": [ 08:58:51 DEBUG }, 08:58:51 DEBUG { 08:58:51 DEBUG "value": [ 08:58:51 DEBUG "metric": { 08:58:51 DEBUG 08:58:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:58:51 DEBUG 08:58:51 DEBUG }, 08:58:51 DEBUG 08:58:51 DEBUG "value": [ 08:58:51 DEBUG [...] 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG [...] 08:58:51 DEBUG "pod": "ds-idrepo-1" 08:58:51 DEBUG 08:58:51 DEBUG }, 08:58:51 DEBUG 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 08:58:51 DEBUG 1684569531, 08:58:51 DEBUG "pod": "ds-cts-0" 08:58:51 DEBUG "0" 08:58:51 DEBUG }, 08:58:51 DEBUG ] 08:58:51 DEBUG "value": [ 08:58:51 DEBUG } 08:58:51 DEBUG 1684569531, 08:58:51 DEBUG ] 08:58:51 DEBUG "7490.546559301536" 08:58:51 DEBUG } 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG } 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG } 08:58:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569531" 08:58:51 INFO [http_cmd]: http status code OK 08:58:51 DEBUG --- status code --- 08:58:51 DEBUG http status code is 200 (expected 200) 08:58:51 DEBUG --- http response --- 08:58:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:58:51 DEBUG ----- output ----- 08:58:51 DEBUG { 08:58:51 DEBUG "status": "success", 08:58:51 DEBUG "data": { 08:58:51 DEBUG "resultType": "vector", 08:58:51 DEBUG "result": [ 08:58:51 DEBUG { 08:58:51 DEBUG "metric": { 08:58:51 DEBUG "pod": "ds-idrepo-2" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG [...] 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG "pod": "ds-idrepo-1" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 1684569531, 08:58:51 DEBUG "0" 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG } 08:58:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569531" 08:58:51 INFO [http_cmd]: http status code OK 08:58:51 DEBUG --- status code --- 08:58:51 DEBUG http status code is 200 (expected 200) 08:58:51 DEBUG --- http response --- 08:58:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:58:51 DEBUG ----- output ----- 08:58:51 DEBUG { 08:58:51 DEBUG "status": "success", 08:58:51 DEBUG "data": { 08:58:51 DEBUG "resultType": "vector", 08:58:51 DEBUG "result": [ 08:58:51 DEBUG { 08:58:51 DEBUG "metric": { 08:58:51 DEBUG "pod": "ds-idrepo-2" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG [...] 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG "pod": "ds-idrepo-1" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 1684569531, 08:58:51 DEBUG "0" 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG } 08:58:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569531" 08:58:51 INFO [http_cmd]: http status code OK 08:58:51 DEBUG --- status code --- 08:58:51 DEBUG http status code is 200 (expected 200) 08:58:51 DEBUG --- http response --- 08:58:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:58:51 DEBUG ----- output ----- 08:58:51 DEBUG { 08:58:51 DEBUG "status": "success", 08:58:51 DEBUG "data": { 08:58:51 DEBUG "resultType": "vector", 08:58:51 DEBUG "result": [ 08:58:51 DEBUG { 08:58:51 DEBUG "metric": { 08:58:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG [...] 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 1684569531, 08:58:51 DEBUG "0" 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG } 08:58:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569531" 08:58:51 INFO [http_cmd]: http status code OK 08:58:51 DEBUG --- status code --- 08:58:51 DEBUG http status code is 200 (expected 200) 08:58:51 DEBUG --- http response --- 08:58:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:51 DEBUG ----- output ----- 08:58:51 DEBUG { 08:58:51 DEBUG "status": "success", 08:58:51 DEBUG "data": { 08:58:51 DEBUG "resultType": "vector", 08:58:51 DEBUG "result": [ 08:58:51 DEBUG { 08:58:51 DEBUG "metric": { 08:58:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG [...] 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG 08:58:51 DEBUG "pod": "ds-cts-0" 08:58:51 DEBUG }, 08:58:51 DEBUG "value": [ 08:58:51 DEBUG 1684569531, 08:58:51 DEBUG "17.255110167241835" 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG ] 08:58:51 DEBUG } 08:58:51 DEBUG } 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569532" 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "pod": "ds-cts-2" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "pod": "ds-idrepo-1" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "0" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569532" 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "pod": "ds-cts-2" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "pod": "ds-idrepo-1" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "0" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569532" 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569532" 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "14220263424" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "0.2" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569532" 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "instance": "10.142.0.17:9100" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "instance": "10.142.0.61:9100" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "6587.133333333333" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569532" 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569532" 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569532" 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569532" 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG { 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG "data": { 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "data": { 08:58:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG }, 08:58:52 DEBUG "result": [ 08:58:52 DEBUG "value": [ 08:58:52 DEBUG { 08:58:52 DEBUG 08:58:52 DEBUG "metric": { 08:58:52 DEBUG 08:58:52 DEBUG "pod": "ds-idrepo-2" 08:58:52 DEBUG 08:58:52 DEBUG }, 08:58:52 DEBUG [...] 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:58:52 DEBUG [...] 08:58:52 DEBUG }, 08:58:52 DEBUG 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG 08:58:52 DEBUG "0" 08:58:52 DEBUG "pod": "ds-idrepo-1" 08:58:52 DEBUG ] 08:58:52 DEBUG }, 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG } 08:58:52 DEBUG "value": [ 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG ] 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG } 08:58:52 DEBUG "0" 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG } 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG ] 08:58:52 DEBUG { 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG } 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG } 08:58:52 DEBUG "data": { 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "data": { 08:58:52 DEBUG "instance": "10.142.0.17:9100" 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG }, 08:58:52 DEBUG "result": [ 08:58:52 DEBUG "value": [ 08:58:52 DEBUG { 08:58:52 DEBUG 08:58:52 DEBUG "metric": { 08:58:52 DEBUG 08:58:52 DEBUG "instance": "10.142.0.17:9100" 08:58:52 DEBUG 08:58:52 DEBUG }, 08:58:52 DEBUG [...] 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "instance": "10.142.0.61:9100" 08:58:52 DEBUG [...] 08:58:52 DEBUG }, 08:58:52 DEBUG 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG 08:58:52 DEBUG "0.00043333333333445503" 08:58:52 DEBUG "instance": "10.142.0.61:9100" 08:58:52 DEBUG ] 08:58:52 DEBUG }, 08:58:52 DEBUG } 08:58:52 DEBUG "value": [ 08:58:52 DEBUG ] 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG } 08:58:52 DEBUG "0" 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569532" 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "0" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569532" 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "instance": "10.142.0.17:9100" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "instance": "10.142.0.61:9100" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "2803.0666666666666" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569532" 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569532" 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "pod": "ds-cts-0" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "0.8258856148423444" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "pod": "ds-cts-0" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "0.09088638731101653" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 08:58:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569532" 08:58:52 INFO [http_cmd]: http status code OK 08:58:52 DEBUG --- status code --- 08:58:52 DEBUG http status code is 200 (expected 200) 08:58:52 DEBUG --- http response --- 08:58:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:58:52 DEBUG ----- output ----- 08:58:52 DEBUG { 08:58:52 DEBUG "status": "success", 08:58:52 DEBUG "data": { 08:58:52 DEBUG "resultType": "vector", 08:58:52 DEBUG "result": [ 08:58:52 DEBUG { 08:58:52 DEBUG "metric": { 08:58:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG [...] 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG 08:58:52 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:58:52 DEBUG }, 08:58:52 DEBUG "value": [ 08:58:52 DEBUG 1684569532, 08:58:52 DEBUG "1.0269101602066921" 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG ] 08:58:52 DEBUG } 08:58:52 DEBUG } 127.0.0.1 - - [20/May/2023 08:58:55] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:59:01] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:59:08] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:59:15] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:59:21] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 08:59:28] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:59:29 INFO 08:59:29 INFO [loop_until]: kubectl --namespace=xlou top pods 08:59:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 08:59:29 INFO [loop_until]: OK (rc = 0) 08:59:29 DEBUG --- stdout --- 08:59:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 11m 9404Mi am-7849cf7bdb-lxfkl 10m 9340Mi am-7849cf7bdb-td7js 12m 9400Mi ds-cts-0 68m 13293Mi ds-cts-1 254m 13395Mi ds-cts-2 49m 13486Mi ds-idrepo-0 17m 13662Mi ds-idrepo-1 13m 13555Mi ds-idrepo-2 16m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 4m 3297Mi lodemon-58bb5f9bf4-7mmfr 36m 77Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1019m 4991Mi 08:59:29 DEBUG --- stderr --- 08:59:29 DEBUG 127.0.0.1 - - [20/May/2023 08:59:35] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:59:41 INFO 08:59:41 INFO [loop_until]: kubectl --namespace=xlou top node 08:59:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 127.0.0.1 - - [20/May/2023 08:59:41] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:59:41 INFO [loop_until]: OK (rc = 0) 08:59:41 DEBUG --- stdout --- 08:59:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 74m 0% 11553Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 75m 0% 4954Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 95m 0% 11677Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 86m 0% 12009Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 155m 0% 3144Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 124m 0% 1367Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 5299Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14654Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 155m 0% 14777Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 70m 0% 14778Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 66m 0% 14585Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 140m 0% 14849Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 129m 0% 14672Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1081m 6% 7117Mi 12% 08:59:41 DEBUG --- stderr --- 08:59:41 DEBUG 127.0.0.1 - - [20/May/2023 08:59:48] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 08:59:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569591" 08:59:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569591" 08:59:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569591" 08:59:51 INFO [http_cmd]: http status code OK 08:59:51 DEBUG --- status code --- 08:59:51 DEBUG http status code is 200 (expected 200) 08:59:51 DEBUG --- http response --- 08:59:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 08:59:51 DEBUG ----- output ----- 08:59:51 DEBUG { 08:59:51 DEBUG "status": "success", 08:59:51 DEBUG "data": { 08:59:51 DEBUG "resultType": "vector", 08:59:51 DEBUG "result": [ 08:59:51 DEBUG { 08:59:51 DEBUG "metric": { 08:59:51 DEBUG "pod": "ds-cts-2" 08:59:51 DEBUG }, 08:59:51 DEBUG "value": [ 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG [...] 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG "pod": "ds-idrepo-1" 08:59:51 DEBUG }, 08:59:51 DEBUG "value": [ 08:59:51 DEBUG 1684569591, 08:59:51 DEBUG "0" 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG } 08:59:51 INFO [http_cmd]: http status code OK 08:59:51 DEBUG --- status code --- 08:59:51 DEBUG http status code is 200 (expected 200) 08:59:51 DEBUG --- http response --- 08:59:51 INFO [http_cmd]: http status code OK 08:59:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:59:51 DEBUG --- status code --- 08:59:51 DEBUG ----- output ----- 08:59:51 DEBUG http status code is 200 (expected 200) 08:59:51 DEBUG { 08:59:51 DEBUG --- http response --- 08:59:51 DEBUG "status": "success", 08:59:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:59:51 DEBUG "data": { 08:59:51 DEBUG ----- output ----- 08:59:51 DEBUG "resultType": "vector", 08:59:51 DEBUG { 08:59:51 DEBUG "result": [ 08:59:51 DEBUG "status": "success", 08:59:51 DEBUG { 08:59:51 DEBUG "data": { 08:59:51 DEBUG "metric": { 08:59:51 DEBUG "resultType": "vector", 08:59:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:59:51 DEBUG "result": [ 08:59:51 DEBUG }, 08:59:51 DEBUG { 08:59:51 DEBUG "value": [ 08:59:51 DEBUG "metric": { 08:59:51 DEBUG 08:59:51 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:59:51 DEBUG 08:59:51 DEBUG }, 08:59:51 DEBUG 08:59:51 DEBUG "value": [ 08:59:51 DEBUG [...] 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG [...] 08:59:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:59:51 DEBUG 08:59:51 DEBUG }, 08:59:51 DEBUG 08:59:51 DEBUG "value": [ 08:59:51 DEBUG 08:59:51 DEBUG 1684569591, 08:59:51 DEBUG "pod": "ds-idrepo-1" 08:59:51 DEBUG "0" 08:59:51 DEBUG }, 08:59:51 DEBUG ] 08:59:51 DEBUG "value": [ 08:59:51 DEBUG } 08:59:51 DEBUG 1684569591, 08:59:51 DEBUG ] 08:59:51 DEBUG "0" 08:59:51 DEBUG } 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG } 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG } 08:59:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569591" 08:59:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569591" 08:59:51 INFO [http_cmd]: http status code OK 08:59:51 DEBUG --- status code --- 08:59:51 DEBUG http status code is 200 (expected 200) 08:59:51 DEBUG --- http response --- 08:59:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:59:51 DEBUG ----- output ----- 08:59:51 DEBUG { 08:59:51 DEBUG "status": "success", 08:59:51 DEBUG "data": { 08:59:51 DEBUG "resultType": "vector", 08:59:51 DEBUG "result": [ 08:59:51 DEBUG { 08:59:51 DEBUG "metric": { 08:59:51 DEBUG "pod": "ds-idrepo-2" 08:59:51 DEBUG }, 08:59:51 DEBUG "value": [ 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG [...] 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG "pod": "ds-idrepo-1" 08:59:51 DEBUG }, 08:59:51 DEBUG "value": [ 08:59:51 DEBUG 1684569591, 08:59:51 DEBUG "0" 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG } 08:59:51 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569591" 08:59:51 INFO [http_cmd]: http status code OK 08:59:51 DEBUG --- status code --- 08:59:51 DEBUG http status code is 200 (expected 200) 08:59:51 DEBUG --- http response --- 08:59:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:59:51 DEBUG ----- output ----- 08:59:51 DEBUG { 08:59:51 DEBUG "status": "success", 08:59:51 DEBUG "data": { 08:59:51 DEBUG "resultType": "vector", 08:59:51 DEBUG "result": [ 08:59:51 DEBUG { 08:59:51 DEBUG "metric": { 08:59:51 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:59:51 DEBUG }, 08:59:51 DEBUG "value": [ 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG [...] 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:59:51 DEBUG }, 08:59:51 DEBUG "value": [ 08:59:51 DEBUG 1684569591, 08:59:51 DEBUG "0" 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG } 08:59:51 INFO [http_cmd]: http status code OK 08:59:51 DEBUG --- status code --- 08:59:51 DEBUG http status code is 200 (expected 200) 08:59:51 DEBUG --- http response --- 08:59:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:59:51 DEBUG ----- output ----- 08:59:51 DEBUG { 08:59:51 DEBUG "status": "success", 08:59:51 DEBUG "data": { 08:59:51 DEBUG "resultType": "vector", 08:59:51 DEBUG "result": [ 08:59:51 DEBUG { 08:59:51 DEBUG "metric": { 08:59:51 DEBUG "pod": "ds-idrepo-2" 08:59:51 DEBUG }, 08:59:51 DEBUG "value": [ 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG [...] 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG 08:59:51 DEBUG "pod": "ds-idrepo-1" 08:59:51 DEBUG }, 08:59:51 DEBUG "value": [ 08:59:51 DEBUG 1684569591, 08:59:51 DEBUG "0" 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG ] 08:59:51 DEBUG } 08:59:51 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "0" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "pod": "ds-cts-2" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "pod": "ds-idrepo-1" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "0" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684569592" 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "pod": "ds-cts-2" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "pod": "ds-idrepo-1" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "0" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "14223503360" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "0.2" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "instance": "10.142.0.17:9100" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "instance": "10.142.0.61:9100" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "6778.900000000001" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569592" 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569592" 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569592" 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "data": { 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "pod": "ds-idrepo-2" 08:59:52 DEBUG "result": [ 08:59:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:59:52 DEBUG }, 08:59:52 DEBUG { 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG "instance": "10.142.0.17:9100" 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG }, 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG "pod": "ds-idrepo-1" 08:59:52 DEBUG 08:59:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:59:52 DEBUG }, 08:59:52 DEBUG 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "instance": "10.142.0.61:9100" 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "0" 08:59:52 DEBUG }, 08:59:52 DEBUG "0" 08:59:52 DEBUG ] 08:59:52 DEBUG "value": [ 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG "0" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG ] 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG } 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:59:52 DEBUG } 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "instance": "10.142.0.17:9100" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "instance": "10.142.0.61:9100" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "0.0003333333333330302" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "pod": "am-7849cf7bdb-lxfkl" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "pod": "am-7849cf7bdb-j4rs8" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "0" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "instance": "10.142.0.17:9100" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "instance": "10.142.0.61:9100" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "3170.266666666667" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569592" 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "pod": "idm-78f4b47cb9-fmzjg" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "0" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "pod": "admin-ui-8666f85968-sjb9j" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "pod": "ds-idrepo-1" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "0" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 08:59:52 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684569592" 08:59:52 INFO [http_cmd]: http status code OK 08:59:52 DEBUG --- status code --- 08:59:52 DEBUG http status code is 200 (expected 200) 08:59:52 DEBUG --- http response --- 08:59:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 08:59:52 DEBUG ----- output ----- 08:59:52 DEBUG { 08:59:52 DEBUG "status": "success", 08:59:52 DEBUG "data": { 08:59:52 DEBUG "resultType": "vector", 08:59:52 DEBUG "result": [ 08:59:52 DEBUG { 08:59:52 DEBUG "metric": { 08:59:52 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG [...] 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG 08:59:52 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 08:59:52 DEBUG }, 08:59:52 DEBUG "value": [ 08:59:52 DEBUG 1684569592, 08:59:52 DEBUG "1.0311009329707517" 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG ] 08:59:52 DEBUG } 08:59:52 DEBUG } 127.0.0.1 - - [20/May/2023 08:59:55] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 09:00:02] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 09:00:08] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 09:00:15] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 09:00:22] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 09:00:29] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 09:00:29 INFO 09:00:29 INFO [loop_until]: kubectl --namespace=xlou top pods 09:00:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:00:29 INFO [loop_until]: OK (rc = 0) 09:00:29 DEBUG --- stdout --- 09:00:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-sjb9j 1m 5Mi am-7849cf7bdb-j4rs8 12m 9404Mi am-7849cf7bdb-lxfkl 11m 9340Mi am-7849cf7bdb-td7js 12m 9400Mi ds-cts-0 69m 13297Mi ds-cts-1 98m 13399Mi ds-cts-2 103m 13466Mi ds-idrepo-0 17m 13663Mi ds-idrepo-1 12m 13555Mi ds-idrepo-2 15m 13521Mi end-user-ui-787cb4f6b4-s5n85 1m 5Mi idm-78f4b47cb9-7z7ld 6m 3395Mi idm-78f4b47cb9-fmzjg 4m 3297Mi lodemon-58bb5f9bf4-7mmfr 39m 77Mi login-ui-57fddf97c8-82h9f 1m 3Mi overseer-0-6967fc998-8q7zx 1016m 4991Mi 09:00:29 DEBUG --- stderr --- 09:00:29 DEBUG 127.0.0.1 - - [20/May/2023 09:00:35] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 09:00:41 INFO 09:00:41 INFO [loop_until]: kubectl --namespace=xlou top node 09:00:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:00:42 INFO [loop_until]: OK (rc = 0) 09:00:42 DEBUG --- stdout --- 09:00:42 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 74m 0% 11559Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 73m 0% 4954Mi 8% gke-xlou-cdm-default-pool-cab466d1-c9gd 95m 0% 11680Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 90m 0% 12006Mi 20% gke-xlou-cdm-default-pool-cab466d1-ggjd 180m 1% 3055Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 125m 0% 1369Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 94m 0% 5300Mi 9% gke-xlou-cdm-ds-c193de6c-0rg3 64m 0% 14654Mi 24% gke-xlou-cdm-ds-c193de6c-3c57 127m 0% 14782Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 72m 0% 14778Mi 25% gke-xlou-cdm-ds-c193de6c-7dtv 69m 0% 14583Mi 24% gke-xlou-cdm-ds-c193de6c-gqp6 120m 0% 14825Mi 25% gke-xlou-cdm-ds-c193de6c-zp2c 132m 0% 14670Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 649m 4% 6753Mi 11% 09:00:42 DEBUG --- stderr --- 09:00:42 DEBUG 127.0.0.1 - - [20/May/2023 09:00:42] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 09:00:43] "GET /monitoring/average?start_time=23-05-20_01:24:30&stop_time=23-05-20_07:25:58 HTTP/1.1" 200 - 09:00:45 INFO Finished: True 09:00:45 INFO Waiting for threads to register finish flag 09:01:42 INFO Done. Have a nice day! :) 127.0.0.1 - - [20/May/2023 09:01:42] "GET /monitoring/stop HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 09:01:42] "GET /monitoring/average?start_time=23-05-19_19:23:04&stop_time=23-05-20_01:24:30 HTTP/1.1" 200 - 09:01:50 INFO File /tmp/lodemon_data-23-05-19_19:49:37/am_cts_reaper_cache_size.json does not exist. Skipping... 09:01:50 INFO File /tmp/lodemon_data-23-05-19_19:49:37/am_cts_reaper_search_seconds_total.json does not exist. Skipping... 09:01:50 INFO File /tmp/lodemon_data-23-05-19_19:49:37/ds_replication_replica_replayed_updates_conflicts_resolved.json does not exist. Skipping... 09:01:50 INFO File /tmp/lodemon_data-23-05-19_19:49:37/node_disk_read_bytes_total.json does not exist. Skipping... 09:01:50 INFO File /tmp/lodemon_data-23-05-19_19:49:37/node_disk_written_bytes_total.json does not exist. Skipping... 09:01:50 INFO File /tmp/lodemon_data-23-05-19_19:49:37/ds_backend_entry_count.json does not exist. Skipping... 127.0.0.1 - - [20/May/2023 09:01:58] "GET /monitoring/process HTTP/1.1" 200 - 127.0.0.1 - - [20/May/2023 09:01:58] "GET /monitoring/average?start_time=23-05-20_01:24:30&stop_time=23-05-20_07:25:58 HTTP/1.1" 200 -