==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-58f494d89b-c4l9j Namespace: xlou Priority: 0 Node: gke-xlou-cdm-default-pool-cab466d1-h89r/10.142.0.26 Start Time: Thu, 18 May 2023 16:59:53 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=58f494d89b skaffold.dev/run-id=ef05c827-531e-4509-8cc3-e4b188bed3e6 Annotations: Status: Running IP: 10.106.40.190 IPs: IP: 10.106.40.190 Controlled By: ReplicaSet/lodemon-58f494d89b Containers: lodemon: Container ID: containerd://924d880506061667bae8f96060afde9499141e0b70fcad1f064f87161086925c Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:21aa7819571d93a7f3c2e1355b96b08ae4717ae2 Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:b4cd4ad79a4fe740bf283ecfb348b507abe0e3672cdd540fd5aedaa47acc0a7b Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Thu, 18 May 2023 17:00:38 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ffwxj (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-ffwxj: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 22:56:42 INFO 22:56:42 INFO [loop_until]: kubectl --namespace=xlou top pods 22:56:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:56:42 INFO [loop_until]: OK (rc = 0) 22:56:42 DEBUG --- stdout --- 22:56:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 3985m 8983Mi am-7849cf7bdb-vcmhd 4241m 8975Mi am-7849cf7bdb-vnjrd 4166m 8955Mi ds-cts-0 5032m 13867Mi ds-cts-1 4900m 13840Mi ds-cts-2 4617m 13824Mi ds-idrepo-0 2733m 13823Mi ds-idrepo-1 12m 13826Mi ds-idrepo-2 19m 13822Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 5m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 4m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1247m 563Mi 22:56:42 DEBUG --- stderr --- 22:56:42 DEBUG 22:56:45 INFO 22:56:45 INFO [loop_until]: kubectl --namespace=xlou top node 22:56:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:56:46 INFO [loop_until]: OK (rc = 0) 22:56:46 DEBUG --- stdout --- 22:56:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4557m 28% 11153Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 3839Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 4988m 31% 11210Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7507m 47% 11534Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 179m 1% 3062Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 96m 0% 1247Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 4949Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14806Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 5304m 33% 14715Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 4729m 29% 14618Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 73m 0% 14777Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 4989m 31% 14634Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2618m 16% 14813Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1315m 8% 2584Mi 4% 22:56:46 DEBUG --- stderr --- 22:56:46 DEBUG 22:57:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447049" 22:57:29 INFO [http_cmd]: http status code OK 22:57:29 DEBUG --- status code --- 22:57:29 DEBUG http status code is 200 (expected 200) 22:57:29 DEBUG --- http response --- 22:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:57:29 DEBUG ----- output ----- 22:57:29 DEBUG { 22:57:29 DEBUG "status": "success", 22:57:29 DEBUG "data": { 22:57:29 DEBUG "resultType": "vector", 22:57:29 DEBUG "result": [ 22:57:29 DEBUG { 22:57:29 DEBUG "metric": { 22:57:29 DEBUG "pod": "ds-cts-1" 22:57:29 DEBUG }, 22:57:29 DEBUG "value": [ 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG [...] 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG "pod": "ds-idrepo-1" 22:57:29 DEBUG }, 22:57:29 DEBUG "value": [ 22:57:29 DEBUG 1684447049, 22:57:29 DEBUG "0" 22:57:29 DEBUG ] 22:57:29 DEBUG } 22:57:29 DEBUG ] 22:57:29 DEBUG } 22:57:29 DEBUG } 22:57:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447049" 22:57:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447049" 22:57:29 INFO [http_cmd]: http status code OK 22:57:29 DEBUG --- status code --- 22:57:29 DEBUG http status code is 200 (expected 200) 22:57:29 DEBUG --- http response --- 22:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:29 DEBUG ----- output ----- 22:57:29 DEBUG { 22:57:29 DEBUG "status": "success", 22:57:29 DEBUG "data": { 22:57:29 DEBUG "resultType": "vector", 22:57:29 DEBUG "result": [ 22:57:29 DEBUG { 22:57:29 DEBUG "metric": { 22:57:29 DEBUG "pod": "ds-cts-1" 22:57:29 DEBUG }, 22:57:29 DEBUG "value": [ 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG [...] 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG "pod": "ds-idrepo-1" 22:57:29 DEBUG }, 22:57:29 DEBUG "value": [ 22:57:29 DEBUG 1684447049, 22:57:29 DEBUG "0" 22:57:29 DEBUG ] 22:57:29 DEBUG } 22:57:29 DEBUG ] 22:57:29 DEBUG } 22:57:29 DEBUG } 22:57:29 INFO [http_cmd]: http status code OK 22:57:29 DEBUG --- status code --- 22:57:29 DEBUG http status code is 200 (expected 200) 22:57:29 DEBUG --- http response --- 22:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:29 DEBUG ----- output ----- 22:57:29 DEBUG { 22:57:29 DEBUG "status": "success", 22:57:29 DEBUG "data": { 22:57:29 DEBUG "resultType": "vector", 22:57:29 DEBUG "result": [ 22:57:29 DEBUG { 22:57:29 DEBUG "metric": { 22:57:29 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:57:29 DEBUG }, 22:57:29 DEBUG "value": [ 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG [...] 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:57:29 DEBUG }, 22:57:29 DEBUG "value": [ 22:57:29 DEBUG 1684447049, 22:57:29 DEBUG "209.46666666666667" 22:57:29 DEBUG ] 22:57:29 DEBUG } 22:57:29 DEBUG ] 22:57:29 DEBUG } 22:57:29 DEBUG } 22:57:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447049" 22:57:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447049" 22:57:29 INFO [http_cmd]: http status code OK 22:57:29 DEBUG --- status code --- 22:57:29 DEBUG http status code is 200 (expected 200) 22:57:29 DEBUG --- http response --- 22:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:57:29 DEBUG ----- output ----- 22:57:29 DEBUG { 22:57:29 DEBUG "status": "success", 22:57:29 DEBUG "data": { 22:57:29 DEBUG "resultType": "vector", 22:57:29 DEBUG "result": [ 22:57:29 DEBUG { 22:57:29 DEBUG "metric": { 22:57:29 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:57:29 DEBUG }, 22:57:29 DEBUG "value": [ 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG [...] 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG 22:57:29 DEBUG "pod": "ds-cts-2" 22:57:29 DEBUG }, 22:57:29 DEBUG "value": [ 22:57:29 DEBUG 1684447049, 22:57:29 DEBUG "4.90804632003868" 22:57:29 DEBUG ] 22:57:29 DEBUG } 22:57:29 DEBUG ] 22:57:29 DEBUG } 22:57:29 DEBUG } 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "ds-cts-2" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447049, 22:57:30 DEBUG "28.12131474705936" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "instance": "10.142.0.17:9100" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "instance": "10.142.0.61:9100" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "6992904.933333334" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447050" 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "ds-idrepo-2" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "ds-idrepo-1" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "0" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "ds-idrepo-2" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "ds-idrepo-1" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "0" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "instance": "10.142.0.17:9100" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "instance": "10.142.0.61:9100" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "0.0016666666666660982" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "209.56666666666666" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "ds-cts-1" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "ds-idrepo-1" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "0" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "0.2" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "628.6333333333333" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "instance": "10.142.0.17:9100" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "instance": "10.142.0.61:9100" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "0.0009999999999996826" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "628.6666666666666" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "ds-cts-0" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "0" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "0" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447050" 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447050" 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "pod": "ds-idrepo-2" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "pod": "ds-idrepo-1" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "0" 22:57:30 DEBUG ] 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG ] 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG } 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG } 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 22:57:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 22:57:30 DEBUG }, 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 22:57:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 22:57:30 DEBUG }, 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "14667472896" 22:57:30 DEBUG "1.3192961339989506" 22:57:30 DEBUG ] 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447050" 22:57:30 INFO [http_cmd]: http status code OK 22:57:30 DEBUG --- status code --- 22:57:30 DEBUG http status code is 200 (expected 200) 22:57:30 DEBUG --- http response --- 22:57:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:30 DEBUG ----- output ----- 22:57:30 DEBUG { 22:57:30 DEBUG "status": "success", 22:57:30 DEBUG "data": { 22:57:30 DEBUG "resultType": "vector", 22:57:30 DEBUG "result": [ 22:57:30 DEBUG { 22:57:30 DEBUG "metric": { 22:57:30 DEBUG "instance": "10.142.0.17:9100" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG [...] 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG 22:57:30 DEBUG "instance": "10.142.0.61:9100" 22:57:30 DEBUG }, 22:57:30 DEBUG "value": [ 22:57:30 DEBUG 1684447050, 22:57:30 DEBUG "6996057.066666666" 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG ] 22:57:30 DEBUG } 22:57:30 DEBUG } 22:57:42 INFO 22:57:42 INFO [loop_until]: kubectl --namespace=xlou top pods 22:57:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:57:42 INFO [loop_until]: OK (rc = 0) 22:57:42 DEBUG --- stdout --- 22:57:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 3949m 8967Mi am-7849cf7bdb-vcmhd 4359m 8971Mi am-7849cf7bdb-vnjrd 4211m 8955Mi ds-cts-0 5037m 13935Mi ds-cts-1 4914m 13839Mi ds-cts-2 4670m 13823Mi ds-idrepo-0 2619m 13823Mi ds-idrepo-1 12m 13826Mi ds-idrepo-2 15m 13822Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 4m 3347Mi lodemon-58f494d89b-c4l9j 3m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1177m 564Mi 22:57:42 DEBUG --- stderr --- 22:57:42 DEBUG 22:57:46 INFO 22:57:46 INFO [loop_until]: kubectl --namespace=xlou top node 22:57:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:57:46 INFO [loop_until]: OK (rc = 0) 22:57:46 DEBUG --- stdout --- 22:57:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4519m 28% 11150Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 74m 0% 3837Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 5059m 31% 11206Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7394m 46% 11534Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 174m 1% 3064Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 90m 0% 1250Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 92m 0% 4952Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 65m 0% 14808Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 5038m 31% 14733Mi 25% gke-xlou-cdm-ds-c193de6c-45kq 4723m 29% 14619Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 69m 0% 14774Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 4984m 31% 14633Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2670m 16% 14814Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1286m 8% 2583Mi 4% 22:57:46 DEBUG --- stderr --- 22:57:46 DEBUG 22:58:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447109" 22:58:29 INFO [http_cmd]: http status code OK 22:58:29 DEBUG --- status code --- 22:58:29 DEBUG http status code is 200 (expected 200) 22:58:29 DEBUG --- http response --- 22:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:58:29 DEBUG ----- output ----- 22:58:29 DEBUG { 22:58:29 DEBUG "status": "success", 22:58:29 DEBUG "data": { 22:58:29 DEBUG "resultType": "vector", 22:58:29 DEBUG "result": [ 22:58:29 DEBUG { 22:58:29 DEBUG "metric": { 22:58:29 DEBUG "pod": "ds-cts-1" 22:58:29 DEBUG }, 22:58:29 DEBUG "value": [ 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG [...] 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG "pod": "ds-idrepo-1" 22:58:29 DEBUG }, 22:58:29 DEBUG "value": [ 22:58:29 DEBUG 1684447109, 22:58:29 DEBUG "0" 22:58:29 DEBUG ] 22:58:29 DEBUG } 22:58:29 DEBUG ] 22:58:29 DEBUG } 22:58:29 DEBUG } 22:58:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447109" 22:58:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447109" 22:58:29 INFO [http_cmd]: http status code OK 22:58:29 INFO [http_cmd]: http status code OK 22:58:29 DEBUG --- status code --- 22:58:29 DEBUG --- status code --- 22:58:29 DEBUG http status code is 200 (expected 200) 22:58:29 DEBUG http status code is 200 (expected 200) 22:58:29 DEBUG --- http response --- 22:58:29 DEBUG --- http response --- 22:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:29 DEBUG ----- output ----- 22:58:29 DEBUG ----- output ----- 22:58:29 DEBUG { 22:58:29 DEBUG { 22:58:29 DEBUG "status": "success", 22:58:29 DEBUG "status": "success", 22:58:29 DEBUG "data": { 22:58:29 DEBUG "data": { 22:58:29 DEBUG "resultType": "vector", 22:58:29 DEBUG "resultType": "vector", 22:58:29 DEBUG "result": [ 22:58:29 DEBUG "result": [ 22:58:29 DEBUG { 22:58:29 DEBUG { 22:58:29 DEBUG "metric": { 22:58:29 DEBUG "metric": { 22:58:29 DEBUG "pod": "ds-cts-1" 22:58:29 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:58:29 DEBUG }, 22:58:29 DEBUG }, 22:58:29 DEBUG "value": [ 22:58:29 DEBUG "value": [ 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG [...] 22:58:29 DEBUG [...] 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG 22:58:29 DEBUG "pod": "ds-idrepo-1" 22:58:29 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:58:29 DEBUG }, 22:58:29 DEBUG }, 22:58:29 DEBUG "value": [ 22:58:29 DEBUG "value": [ 22:58:29 DEBUG 1684447109, 22:58:29 DEBUG 1684447109, 22:58:29 DEBUG "0" 22:58:29 DEBUG "212.5" 22:58:29 DEBUG ] 22:58:29 DEBUG ] 22:58:29 DEBUG } 22:58:29 DEBUG } 22:58:29 DEBUG ] 22:58:29 DEBUG ] 22:58:29 DEBUG } 22:58:29 DEBUG } 22:58:29 DEBUG } 22:58:29 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "ds-cts-2" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "2.357133805370421" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "ds-cts-2" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "6.4302483192366084" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447110" 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "instance": "10.142.0.17:9100" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "instance": "10.142.0.61:9100" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "0.0015000000000014778" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "ds-idrepo-2" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "ds-idrepo-1" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "0" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447110" 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "instance": "10.142.0.17:9100" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "instance": "10.142.0.61:9100" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "6905866.733333333" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "ds-idrepo-2" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "ds-idrepo-1" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "0" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "ds-cts-1" 22:58:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:58:30 DEBUG }, 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "ds-idrepo-1" 22:58:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:58:30 DEBUG }, 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "0" 22:58:30 DEBUG "212.5" 22:58:30 DEBUG ] 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "0.2" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 DEBUG "instance": "10.142.0.17:9100" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "instance": "10.142.0.61:9100" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "0.0006666666666666524" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "637.4" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "637.4333333333333" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "ds-cts-2" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "3778.9730644111905" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "ds-cts-2" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "0.027957601387985254" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "1.206029201297242" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447110" 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "pod": "ds-idrepo-2" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "pod": "ds-idrepo-1" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "0" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "14589292544" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447110" 22:58:30 INFO [http_cmd]: http status code OK 22:58:30 DEBUG --- status code --- 22:58:30 DEBUG http status code is 200 (expected 200) 22:58:30 DEBUG --- http response --- 22:58:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:30 DEBUG ----- output ----- 22:58:30 DEBUG { 22:58:30 DEBUG "status": "success", 22:58:30 DEBUG "data": { 22:58:30 DEBUG "resultType": "vector", 22:58:30 DEBUG "result": [ 22:58:30 DEBUG { 22:58:30 DEBUG "metric": { 22:58:30 DEBUG "instance": "10.142.0.17:9100" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG [...] 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG 22:58:30 DEBUG "instance": "10.142.0.61:9100" 22:58:30 DEBUG }, 22:58:30 DEBUG "value": [ 22:58:30 DEBUG 1684447110, 22:58:30 DEBUG "6908987.366666667" 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG ] 22:58:30 DEBUG } 22:58:30 DEBUG } 22:58:42 INFO 22:58:42 INFO [loop_until]: kubectl --namespace=xlou top pods 22:58:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:58:42 INFO [loop_until]: OK (rc = 0) 22:58:42 DEBUG --- stdout --- 22:58:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 4024m 8967Mi am-7849cf7bdb-vcmhd 4314m 8971Mi am-7849cf7bdb-vnjrd 4230m 8960Mi ds-cts-0 5128m 13837Mi ds-cts-1 4611m 13840Mi ds-cts-2 4610m 13824Mi ds-idrepo-0 2383m 13822Mi ds-idrepo-1 12m 13826Mi ds-idrepo-2 162m 13822Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 4m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1293m 565Mi 22:58:42 DEBUG --- stderr --- 22:58:42 DEBUG 22:58:46 INFO 22:58:46 INFO [loop_until]: kubectl --namespace=xlou top node 22:58:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:58:46 INFO [loop_until]: OK (rc = 0) 22:58:46 DEBUG --- stdout --- 22:58:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4568m 28% 11155Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 77m 0% 3837Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 4926m 31% 11210Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7465m 46% 11534Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 158m 0% 3053Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 88m 0% 1246Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 92m 0% 4957Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 70m 0% 14807Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 5122m 32% 14638Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 4731m 29% 14616Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 255m 1% 14775Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 5037m 31% 14634Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2676m 16% 14816Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1372m 8% 2583Mi 4% 22:58:46 DEBUG --- stderr --- 22:58:46 DEBUG 22:59:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447169" 22:59:29 INFO [http_cmd]: http status code OK 22:59:29 DEBUG --- status code --- 22:59:29 DEBUG http status code is 200 (expected 200) 22:59:29 DEBUG --- http response --- 22:59:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 22:59:29 DEBUG ----- output ----- 22:59:29 DEBUG { 22:59:29 DEBUG "status": "success", 22:59:29 DEBUG "data": { 22:59:29 DEBUG "resultType": "vector", 22:59:29 DEBUG "result": [ 22:59:29 DEBUG { 22:59:29 DEBUG "metric": { 22:59:29 DEBUG "pod": "ds-cts-1" 22:59:29 DEBUG }, 22:59:29 DEBUG "value": [ 22:59:29 DEBUG 22:59:29 DEBUG 22:59:29 DEBUG 22:59:29 DEBUG [...] 22:59:29 DEBUG 22:59:29 DEBUG 22:59:29 DEBUG 22:59:29 DEBUG "pod": "ds-idrepo-1" 22:59:29 DEBUG }, 22:59:29 DEBUG "value": [ 22:59:29 DEBUG 1684447169, 22:59:29 DEBUG "0" 22:59:29 DEBUG ] 22:59:29 DEBUG } 22:59:29 DEBUG ] 22:59:29 DEBUG } 22:59:29 DEBUG } 22:59:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447169" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "ds-cts-1" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "ds-idrepo-1" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447169, 22:59:30 DEBUG "0" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "213.2" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "ds-cts-0" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447170" 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "instance": "10.142.0.17:9100" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "instance": "10.142.0.61:9100" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "7006532.733333333" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "ds-idrepo-2" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "ds-idrepo-1" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0" 22:59:30 DEBUG ] 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG } 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG ] 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG } 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG } 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "instance": "10.142.0.17:9100" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "instance": "10.142.0.61:9100" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0.001433333333332598" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "ds-idrepo-2" 22:59:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:59:30 DEBUG }, 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "ds-idrepo-1" 22:59:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:59:30 DEBUG }, 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0" 22:59:30 DEBUG "0.2" 22:59:30 DEBUG ] 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "instance": "10.142.0.17:9100" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "instance": "10.142.0.61:9100" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0.0009999999999996236" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "ds-cts-1" 22:59:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:59:30 DEBUG }, 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "ds-idrepo-1" 22:59:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:59:30 DEBUG }, 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0" 22:59:30 DEBUG "213.2" 22:59:30 DEBUG ] 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "639.5666666666667" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "639.7666666666667" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "ds-cts-0" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447170" 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "pod": "ds-idrepo-2" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "pod": "ds-idrepo-1" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "0" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG } 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG } 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "1.317930442220087" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447170" 22:59:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447170" 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "14590509056" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:30 INFO [http_cmd]: http status code OK 22:59:30 DEBUG --- status code --- 22:59:30 DEBUG http status code is 200 (expected 200) 22:59:30 DEBUG --- http response --- 22:59:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:30 DEBUG ----- output ----- 22:59:30 DEBUG { 22:59:30 DEBUG "status": "success", 22:59:30 DEBUG "data": { 22:59:30 DEBUG "resultType": "vector", 22:59:30 DEBUG "result": [ 22:59:30 DEBUG { 22:59:30 DEBUG "metric": { 22:59:30 DEBUG "instance": "10.142.0.17:9100" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG [...] 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG 22:59:30 DEBUG "instance": "10.142.0.61:9100" 22:59:30 DEBUG }, 22:59:30 DEBUG "value": [ 22:59:30 DEBUG 1684447170, 22:59:30 DEBUG "7007838.066666666" 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG ] 22:59:30 DEBUG } 22:59:30 DEBUG } 22:59:42 INFO 22:59:42 INFO [loop_until]: kubectl --namespace=xlou top pods 22:59:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:59:42 INFO [loop_until]: OK (rc = 0) 22:59:42 DEBUG --- stdout --- 22:59:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 4271m 8968Mi am-7849cf7bdb-vcmhd 4489m 8971Mi am-7849cf7bdb-vnjrd 4228m 8960Mi ds-cts-0 4971m 13838Mi ds-cts-1 5438m 13840Mi ds-cts-2 4830m 13824Mi ds-idrepo-0 2832m 13823Mi ds-idrepo-1 12m 13826Mi ds-idrepo-2 18m 13823Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 2m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1254m 566Mi 22:59:42 DEBUG --- stderr --- 22:59:42 DEBUG 22:59:46 INFO 22:59:46 INFO [loop_until]: kubectl --namespace=xlou top node 22:59:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:59:46 INFO [loop_until]: OK (rc = 0) 22:59:46 DEBUG --- stdout --- 22:59:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4754m 29% 11152Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 3841Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 4686m 29% 11207Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7470m 47% 11535Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 164m 1% 3008Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 89m 0% 1246Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 91m 0% 4956Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 63m 0% 14806Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 5209m 32% 14635Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 4726m 29% 14619Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 76m 0% 14776Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 5253m 33% 14639Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2765m 17% 14816Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1362m 8% 2584Mi 4% 22:59:46 DEBUG --- stderr --- 22:59:46 DEBUG 23:00:29 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447229" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "pod": "ds-cts-1" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "ds-idrepo-1" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447229, 23:00:30 DEBUG "0" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG "metric": { 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG }, 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG "value": [ 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:30 DEBUG 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG 23:00:30 DEBUG { 23:00:30 DEBUG 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG [...] 23:00:30 DEBUG "data": { 23:00:30 DEBUG 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG 23:00:30 DEBUG "result": [ 23:00:30 DEBUG 23:00:30 DEBUG { 23:00:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:00:30 DEBUG "metric": { 23:00:30 DEBUG }, 23:00:30 DEBUG "pod": "ds-cts-1" 23:00:30 DEBUG "value": [ 23:00:30 DEBUG }, 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG "211.06666666666666" 23:00:30 DEBUG 23:00:30 DEBUG ] 23:00:30 DEBUG 23:00:30 DEBUG } 23:00:30 DEBUG 23:00:30 DEBUG ] 23:00:30 DEBUG [...] 23:00:30 DEBUG } 23:00:30 DEBUG 23:00:30 DEBUG } 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "ds-idrepo-1" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "ds-idrepo-2" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0.37375844666837743" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "instance": "10.142.0.17:9100" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "instance": "10.142.0.61:9100" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "7016737.366666666" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447230" 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "pod": "ds-idrepo-2" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "ds-idrepo-1" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "instance": "10.142.0.17:9100" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "instance": "10.142.0.61:9100" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0.0013333333333349628" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG { 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG "data": { 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "data": { 23:00:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG }, 23:00:30 DEBUG "result": [ 23:00:30 DEBUG "value": [ 23:00:30 DEBUG { 23:00:30 DEBUG 23:00:30 DEBUG "metric": { 23:00:30 DEBUG 23:00:30 DEBUG "pod": "ds-idrepo-2" 23:00:30 DEBUG 23:00:30 DEBUG }, 23:00:30 DEBUG [...] 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:00:30 DEBUG [...] 23:00:30 DEBUG }, 23:00:30 DEBUG 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG 23:00:30 DEBUG "0.2" 23:00:30 DEBUG "pod": "ds-idrepo-1" 23:00:30 DEBUG ] 23:00:30 DEBUG }, 23:00:30 DEBUG } 23:00:30 DEBUG "value": [ 23:00:30 DEBUG ] 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG } 23:00:30 DEBUG "0" 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "instance": "10.142.0.17:9100" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "instance": "10.142.0.61:9100" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0.0006666666666667708" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG { 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "result": [ 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG { 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "pod": "ds-cts-1" 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:00:30 DEBUG }, 23:00:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:00:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG }, 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG "value": [ 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "ds-idrepo-1" 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:00:30 DEBUG }, 23:00:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:00:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG }, 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0" 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "633.2333333333333" 23:00:30 DEBUG ] 23:00:30 DEBUG "211.06666666666666" 23:00:30 DEBUG "633.1666666666666" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "ds-idrepo-2" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447230" 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447230" 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "pod": "ds-idrepo-2" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "pod": "ds-idrepo-1" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "0" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:00:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:00:30 DEBUG }, 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:00:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:00:30 DEBUG }, 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "1.249679259285797" 23:00:30 DEBUG "14590885888" 23:00:30 DEBUG ] 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447230" 23:00:30 INFO [http_cmd]: http status code OK 23:00:30 DEBUG --- status code --- 23:00:30 DEBUG http status code is 200 (expected 200) 23:00:30 DEBUG --- http response --- 23:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:30 DEBUG ----- output ----- 23:00:30 DEBUG { 23:00:30 DEBUG "status": "success", 23:00:30 DEBUG "data": { 23:00:30 DEBUG "resultType": "vector", 23:00:30 DEBUG "result": [ 23:00:30 DEBUG { 23:00:30 DEBUG "metric": { 23:00:30 DEBUG "instance": "10.142.0.17:9100" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG [...] 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG 23:00:30 DEBUG "instance": "10.142.0.61:9100" 23:00:30 DEBUG }, 23:00:30 DEBUG "value": [ 23:00:30 DEBUG 1684447230, 23:00:30 DEBUG "7020339.366666667" 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG ] 23:00:30 DEBUG } 23:00:30 DEBUG } 23:00:42 INFO 23:00:42 INFO [loop_until]: kubectl --namespace=xlou top pods 23:00:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:00:42 INFO [loop_until]: OK (rc = 0) 23:00:42 DEBUG --- stdout --- 23:00:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 4049m 8968Mi am-7849cf7bdb-vcmhd 4249m 8971Mi am-7849cf7bdb-vnjrd 4170m 8961Mi ds-cts-0 5055m 13838Mi ds-cts-1 4791m 13841Mi ds-cts-2 4558m 13824Mi ds-idrepo-0 2565m 13822Mi ds-idrepo-1 13m 13826Mi ds-idrepo-2 18m 13823Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 1m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1284m 567Mi 23:00:42 DEBUG --- stderr --- 23:00:42 DEBUG 23:00:46 INFO 23:00:46 INFO [loop_until]: kubectl --namespace=xlou top node 23:00:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:00:46 INFO [loop_until]: OK (rc = 0) 23:00:46 DEBUG --- stdout --- 23:00:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4623m 29% 11152Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 74m 0% 3839Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 4950m 31% 11219Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7500m 47% 11533Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 143m 0% 3035Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 95m 0% 1248Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 92m 0% 4957Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14810Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 5210m 32% 14634Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 4860m 30% 14617Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 76m 0% 14776Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 4947m 31% 14638Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2582m 16% 14815Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1410m 8% 2589Mi 4% 23:00:46 DEBUG --- stderr --- 23:00:46 DEBUG 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "ds-cts-1" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "ds-idrepo-1" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "ds-cts-1" 23:01:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:01:30 DEBUG }, 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "ds-idrepo-1" 23:01:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:01:30 DEBUG }, 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0" 23:01:30 DEBUG "213.76666666666665" 23:01:30 DEBUG ] 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "ds-idrepo-0" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "instance": "10.142.0.17:9100" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "instance": "10.142.0.61:9100" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "7035908.2" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "ds-idrepo-2" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "ds-idrepo-1" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "instance": "10.142.0.17:9100" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "instance": "10.142.0.61:9100" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0.0014999999999995833" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "ds-idrepo-2" 23:01:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:01:30 DEBUG }, 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "ds-idrepo-1" 23:01:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:01:30 DEBUG }, 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0" 23:01:30 DEBUG "0.2" 23:01:30 DEBUG ] 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "213.7" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "instance": "10.142.0.17:9100" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "instance": "10.142.0.61:9100" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0.0003333333333333854" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "metric": { 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:01:30 DEBUG "pod": "ds-cts-1" 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG }, 23:01:30 DEBUG }, 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG "value": [ 23:01:30 DEBUG "value": [ 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG { 23:01:30 DEBUG [...] 23:01:30 DEBUG [...] 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "data": { 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "result": [ 23:01:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:01:30 DEBUG "pod": "ds-idrepo-1" 23:01:30 DEBUG { 23:01:30 DEBUG }, 23:01:30 DEBUG }, 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "value": [ 23:01:30 DEBUG "value": [ 23:01:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG }, 23:01:30 DEBUG "641.4666666666667" 23:01:30 DEBUG "0" 23:01:30 DEBUG "value": [ 23:01:30 DEBUG ] 23:01:30 DEBUG ] 23:01:30 DEBUG 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 DEBUG 23:01:30 DEBUG ] 23:01:30 DEBUG ] 23:01:30 DEBUG 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 DEBUG [...] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "641.3666666666667" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447290" 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "14593511424" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "ds-idrepo-0" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "pod": "ds-idrepo-2" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "pod": "ds-idrepo-1" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "0" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "1.2720219727125146" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447290" 23:01:30 INFO [http_cmd]: http status code OK 23:01:30 DEBUG --- status code --- 23:01:30 DEBUG http status code is 200 (expected 200) 23:01:30 DEBUG --- http response --- 23:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:30 DEBUG ----- output ----- 23:01:30 DEBUG { 23:01:30 DEBUG "status": "success", 23:01:30 DEBUG "data": { 23:01:30 DEBUG "resultType": "vector", 23:01:30 DEBUG "result": [ 23:01:30 DEBUG { 23:01:30 DEBUG "metric": { 23:01:30 DEBUG "instance": "10.142.0.17:9100" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG [...] 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG 23:01:30 DEBUG "instance": "10.142.0.61:9100" 23:01:30 DEBUG }, 23:01:30 DEBUG "value": [ 23:01:30 DEBUG 1684447290, 23:01:30 DEBUG "7039162.366666667" 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG ] 23:01:30 DEBUG } 23:01:30 DEBUG } 23:01:42 INFO 23:01:42 INFO [loop_until]: kubectl --namespace=xlou top pods 23:01:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:01:43 INFO [loop_until]: OK (rc = 0) 23:01:43 DEBUG --- stdout --- 23:01:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 4140m 8969Mi am-7849cf7bdb-vcmhd 4268m 8972Mi am-7849cf7bdb-vnjrd 4164m 8961Mi ds-cts-0 5397m 13838Mi ds-cts-1 5067m 13841Mi ds-cts-2 4857m 13847Mi ds-idrepo-0 2459m 13823Mi ds-idrepo-1 13m 13826Mi ds-idrepo-2 15m 13822Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 6m 3347Mi lodemon-58f494d89b-c4l9j 2m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1346m 567Mi 23:01:43 DEBUG --- stderr --- 23:01:43 DEBUG 23:01:46 INFO 23:01:46 INFO [loop_until]: kubectl --namespace=xlou top node 23:01:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:01:46 INFO [loop_until]: OK (rc = 0) 23:01:46 DEBUG --- stdout --- 23:01:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4668m 29% 11154Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 75m 0% 3836Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 4878m 30% 11207Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7345m 46% 11533Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 157m 0% 3058Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 88m 0% 1247Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 95m 0% 4953Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14811Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 5361m 33% 14636Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 5110m 32% 14638Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 68m 0% 14780Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 5226m 32% 14637Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2685m 16% 14825Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1387m 8% 2588Mi 4% 23:01:46 DEBUG --- stderr --- 23:01:46 DEBUG 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "pod": "ds-cts-1" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "pod": "ds-idrepo-1" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "0" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "208.86666666666667" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "pod": "ds-cts-1" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "pod": "ds-idrepo-1" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "0" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:02:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:02:30 DEBUG }, 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "pod": "ds-idrepo-0" 23:02:30 DEBUG "pod": "ds-idrepo-0" 23:02:30 DEBUG }, 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "0.7810558454929527" 23:02:30 DEBUG "2.6189553623033985" 23:02:30 DEBUG ] 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447350" 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG { 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:30 DEBUG "data": { 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG { 23:02:30 DEBUG "result": [ 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG { 23:02:30 DEBUG "data": { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "pod": "ds-idrepo-2" 23:02:30 DEBUG "result": [ 23:02:30 DEBUG }, 23:02:30 DEBUG { 23:02:30 DEBUG "value": [ 23:02:30 DEBUG "metric": { 23:02:30 DEBUG 23:02:30 DEBUG "instance": "10.142.0.17:9100" 23:02:30 DEBUG 23:02:30 DEBUG }, 23:02:30 DEBUG 23:02:30 DEBUG "value": [ 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG "pod": "ds-idrepo-1" 23:02:30 DEBUG 23:02:30 DEBUG }, 23:02:30 DEBUG 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "instance": "10.142.0.61:9100" 23:02:30 DEBUG "0" 23:02:30 DEBUG }, 23:02:30 DEBUG ] 23:02:30 DEBUG "value": [ 23:02:30 DEBUG } 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG ] 23:02:30 DEBUG "6971091.666666667" 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "instance": "10.142.0.17:9100" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "instance": "10.142.0.61:9100" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "0.0015000000000005306" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447350" 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:02:30 DEBUG { 23:02:30 DEBUG { 23:02:30 DEBUG "pod": "ds-idrepo-2" 23:02:30 DEBUG }, 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG "data": { 23:02:30 DEBUG "data": { 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "result": [ 23:02:30 DEBUG "result": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG { 23:02:30 DEBUG { 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:02:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG }, 23:02:30 DEBUG }, 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "value": [ 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "pod": "ds-idrepo-1" 23:02:30 DEBUG }, 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG [...] 23:02:30 DEBUG [...] 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "0.2" 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "0" 23:02:30 DEBUG ] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:02:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG }, 23:02:30 DEBUG }, 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG "value": [ 23:02:30 DEBUG "value": [ 23:02:30 DEBUG } 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "626.5" 23:02:30 DEBUG "208.86666666666667" 23:02:30 DEBUG ] 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "instance": "10.142.0.17:9100" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "instance": "10.142.0.61:9100" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "0.0010000000000001564" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG { 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:30 DEBUG "data": { 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG { 23:02:30 DEBUG "result": [ 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG { 23:02:30 DEBUG "data": { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "pod": "ds-cts-1" 23:02:30 DEBUG "result": [ 23:02:30 DEBUG }, 23:02:30 DEBUG { 23:02:30 DEBUG "value": [ 23:02:30 DEBUG "metric": { 23:02:30 DEBUG 23:02:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:02:30 DEBUG 23:02:30 DEBUG }, 23:02:30 DEBUG 23:02:30 DEBUG "value": [ 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG "pod": "ds-idrepo-1" 23:02:30 DEBUG 23:02:30 DEBUG }, 23:02:30 DEBUG 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:02:30 DEBUG "0" 23:02:30 DEBUG }, 23:02:30 DEBUG ] 23:02:30 DEBUG "value": [ 23:02:30 DEBUG } 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG ] 23:02:30 DEBUG "626.5" 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:02:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:02:30 DEBUG }, 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "pod": "ds-idrepo-0" 23:02:30 DEBUG "pod": "ds-idrepo-0" 23:02:30 DEBUG }, 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "15414.35012603401" 23:02:30 DEBUG "23.1476550573366" 23:02:30 DEBUG ] 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "14593019904" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "pod": "ds-idrepo-2" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "pod": "ds-idrepo-1" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "0" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "1.2784213602368661" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447350" 23:02:30 INFO [http_cmd]: http status code OK 23:02:30 DEBUG --- status code --- 23:02:30 DEBUG http status code is 200 (expected 200) 23:02:30 DEBUG --- http response --- 23:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:30 DEBUG ----- output ----- 23:02:30 DEBUG { 23:02:30 DEBUG "status": "success", 23:02:30 DEBUG "data": { 23:02:30 DEBUG "resultType": "vector", 23:02:30 DEBUG "result": [ 23:02:30 DEBUG { 23:02:30 DEBUG "metric": { 23:02:30 DEBUG "instance": "10.142.0.17:9100" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG [...] 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG 23:02:30 DEBUG "instance": "10.142.0.61:9100" 23:02:30 DEBUG }, 23:02:30 DEBUG "value": [ 23:02:30 DEBUG 1684447350, 23:02:30 DEBUG "6974114.166666666" 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG ] 23:02:30 DEBUG } 23:02:30 DEBUG } 23:02:43 INFO 23:02:43 INFO [loop_until]: kubectl --namespace=xlou top pods 23:02:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:02:43 INFO [loop_until]: OK (rc = 0) 23:02:43 DEBUG --- stdout --- 23:02:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 3964m 8968Mi am-7849cf7bdb-vcmhd 4210m 8972Mi am-7849cf7bdb-vnjrd 4122m 8962Mi ds-cts-0 5409m 13840Mi ds-cts-1 5516m 13841Mi ds-cts-2 5041m 13824Mi ds-idrepo-0 2448m 13823Mi ds-idrepo-1 13m 13826Mi ds-idrepo-2 18m 13823Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 4m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1250m 568Mi 23:02:43 DEBUG --- stderr --- 23:02:43 DEBUG 23:02:46 INFO 23:02:46 INFO [loop_until]: kubectl --namespace=xlou top node 23:02:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:02:46 INFO [loop_until]: OK (rc = 0) 23:02:46 DEBUG --- stdout --- 23:02:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4595m 28% 11156Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 76m 0% 3835Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 4723m 29% 11201Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7469m 47% 11535Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 160m 1% 3059Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 82m 0% 1244Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 91m 0% 4952Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14807Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 5486m 34% 14647Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 5092m 32% 14618Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 76m 0% 14781Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 5897m 37% 14658Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2553m 16% 14813Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1338m 8% 2598Mi 4% 23:02:46 DEBUG --- stderr --- 23:02:46 DEBUG 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "ds-cts-1" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "ds-idrepo-1" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "0" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:03:30 DEBUG "pod": "ds-cts-1" 23:03:30 DEBUG }, 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:03:30 DEBUG "pod": "ds-idrepo-1" 23:03:30 DEBUG }, 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "208.46666666666667" 23:03:30 DEBUG "0" 23:03:30 DEBUG ] 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:03:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:03:30 DEBUG }, 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "ds-cts-2" 23:03:30 DEBUG "pod": "ds-cts-2" 23:03:30 DEBUG }, 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "3.334855198753809" 23:03:30 DEBUG "20.302811229577358" 23:03:30 DEBUG ] 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG { 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG "data": { 23:03:30 DEBUG { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG "data": { 23:03:30 DEBUG { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "result": [ 23:03:30 DEBUG "pod": "ds-idrepo-2" 23:03:30 DEBUG { 23:03:30 DEBUG }, 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "value": [ 23:03:30 DEBUG "instance": "10.142.0.17:9100" 23:03:30 DEBUG 23:03:30 DEBUG }, 23:03:30 DEBUG 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "ds-idrepo-1" 23:03:30 DEBUG 23:03:30 DEBUG }, 23:03:30 DEBUG 23:03:30 DEBUG "value": [ 23:03:30 DEBUG "instance": "10.142.0.61:9100" 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG }, 23:03:30 DEBUG "0" 23:03:30 DEBUG "value": [ 23:03:30 DEBUG ] 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG } 23:03:30 DEBUG "6953952.6" 23:03:30 DEBUG ] 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "ds-idrepo-2" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "ds-idrepo-1" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "0" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "instance": "10.142.0.17:9100" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "instance": "10.142.0.61:9100" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "0.0015333333333330756" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "0.2" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "625.4" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "instance": "10.142.0.17:9100" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "instance": "10.142.0.61:9100" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "0.0003333333333333854" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG } 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG } 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "208.43333333333334" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "625.4333333333333" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "ds-cts-1" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "ds-idrepo-1" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "0" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "ds-cts-2" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "1334.6717189537471" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "ds-cts-2" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "96.0498427552351" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "14593507328" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "pod": "ds-idrepo-2" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "pod": "ds-idrepo-1" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "0" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "1.2684413845861127" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447410" 23:03:30 INFO [http_cmd]: http status code OK 23:03:30 DEBUG --- status code --- 23:03:30 DEBUG http status code is 200 (expected 200) 23:03:30 DEBUG --- http response --- 23:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:30 DEBUG ----- output ----- 23:03:30 DEBUG { 23:03:30 DEBUG "status": "success", 23:03:30 DEBUG "data": { 23:03:30 DEBUG "resultType": "vector", 23:03:30 DEBUG "result": [ 23:03:30 DEBUG { 23:03:30 DEBUG "metric": { 23:03:30 DEBUG "instance": "10.142.0.17:9100" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG [...] 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG 23:03:30 DEBUG "instance": "10.142.0.61:9100" 23:03:30 DEBUG }, 23:03:30 DEBUG "value": [ 23:03:30 DEBUG 1684447410, 23:03:30 DEBUG "6956954.033333333" 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG ] 23:03:30 DEBUG } 23:03:30 DEBUG } 23:03:43 INFO 23:03:43 INFO [loop_until]: kubectl --namespace=xlou top pods 23:03:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:03:43 INFO [loop_until]: OK (rc = 0) 23:03:43 DEBUG --- stdout --- 23:03:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 3863m 8968Mi am-7849cf7bdb-vcmhd 4164m 8973Mi am-7849cf7bdb-vnjrd 3934m 8962Mi ds-cts-0 4845m 13840Mi ds-cts-1 5009m 13841Mi ds-cts-2 4721m 13825Mi ds-idrepo-0 2342m 13822Mi ds-idrepo-1 13m 13826Mi ds-idrepo-2 18m 13823Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 9m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 4m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1191m 568Mi 23:03:43 DEBUG --- stderr --- 23:03:43 DEBUG 23:03:46 INFO 23:03:46 INFO [loop_until]: kubectl --namespace=xlou top node 23:03:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:03:46 INFO [loop_until]: OK (rc = 0) 23:03:46 DEBUG --- stdout --- 23:03:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4373m 27% 11157Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 82m 0% 3835Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 4851m 30% 11210Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7335m 46% 11541Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 162m 1% 3060Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 88m 0% 1249Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 94m 0% 4951Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 65m 0% 14810Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 4940m 31% 14636Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 4769m 30% 14617Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 77m 0% 14777Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 4877m 30% 14646Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2332m 14% 14814Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1290m 8% 2592Mi 4% 23:03:46 DEBUG --- stderr --- 23:03:46 DEBUG 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "ds-cts-1" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-idrepo-1" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "ds-cts-1" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-idrepo-1" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "207.9" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-cts-2" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-cts-0" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "10.469453224008674" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "ds-idrepo-2" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-idrepo-1" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0.2" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "instance": "10.142.0.17:9100" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "instance": "10.142.0.61:9100" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "6985318.833333333" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "ds-idrepo-2" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-idrepo-1" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "instance": "10.142.0.17:9100" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "instance": "10.142.0.61:9100" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0.0015666666666675154" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:30 DEBUG { 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG { 23:04:30 DEBUG "data": { 23:04:30 DEBUG "status": "success", 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "data": { 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG "result": [ 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG { 23:04:30 DEBUG "result": [ 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG "metric": { 23:04:30 DEBUG { 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:04:30 DEBUG "metric": { 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG }, 23:04:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:04:30 DEBUG { 23:04:30 DEBUG "value": [ 23:04:30 DEBUG }, 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG 23:04:30 DEBUG "value": [ 23:04:30 DEBUG "data": { 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "result": [ 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG { 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG "metric": { 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "instance": "10.142.0.17:9100" 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG }, 23:04:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:04:30 DEBUG 23:04:30 DEBUG "value": [ 23:04:30 DEBUG }, 23:04:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:04:30 DEBUG 23:04:30 DEBUG "value": [ 23:04:30 DEBUG }, 23:04:30 DEBUG 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG "207.9" 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG [...] 23:04:30 DEBUG ] 23:04:30 DEBUG "623.6666666666666" 23:04:30 DEBUG 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG "instance": "10.142.0.61:9100" 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 DEBUG }, 23:04:30 DEBUG } 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0.0010000000000002748" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "623.7666666666667" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "ds-cts-1" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-idrepo-1" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-cts-0" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "51.82954590566931" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447470" 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "14605660160" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-cts-2" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "pod": "ds-idrepo-2" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447470" 23:04:30 DEBUG 23:04:30 DEBUG "pod": "ds-idrepo-1" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "0" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447470" 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "1.2851800090624053" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:30 INFO [http_cmd]: http status code OK 23:04:30 DEBUG --- status code --- 23:04:30 DEBUG http status code is 200 (expected 200) 23:04:30 DEBUG --- http response --- 23:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:30 DEBUG ----- output ----- 23:04:30 DEBUG { 23:04:30 DEBUG "status": "success", 23:04:30 DEBUG "data": { 23:04:30 DEBUG "resultType": "vector", 23:04:30 DEBUG "result": [ 23:04:30 DEBUG { 23:04:30 DEBUG "metric": { 23:04:30 DEBUG "instance": "10.142.0.17:9100" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG [...] 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG 23:04:30 DEBUG "instance": "10.142.0.61:9100" 23:04:30 DEBUG }, 23:04:30 DEBUG "value": [ 23:04:30 DEBUG 1684447470, 23:04:30 DEBUG "6986528.033333333" 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG ] 23:04:30 DEBUG } 23:04:30 DEBUG } 23:04:43 INFO 23:04:43 INFO [loop_until]: kubectl --namespace=xlou top pods 23:04:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:04:43 INFO [loop_until]: OK (rc = 0) 23:04:43 DEBUG --- stdout --- 23:04:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 4197m 8968Mi am-7849cf7bdb-vcmhd 4296m 8973Mi am-7849cf7bdb-vnjrd 4228m 8962Mi ds-cts-0 5406m 13840Mi ds-cts-1 5996m 13841Mi ds-cts-2 4839m 13824Mi ds-idrepo-0 2527m 13822Mi ds-idrepo-1 12m 13826Mi ds-idrepo-2 19m 13823Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 7m 2478Mi idm-78f4b47cb9-gv5tc 10m 3347Mi lodemon-58f494d89b-c4l9j 4m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1268m 569Mi 23:04:43 DEBUG --- stderr --- 23:04:43 DEBUG 23:04:46 INFO 23:04:46 INFO [loop_until]: kubectl --namespace=xlou top node 23:04:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:04:47 INFO [loop_until]: OK (rc = 0) 23:04:47 DEBUG --- stdout --- 23:04:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 4797m 30% 11155Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 79m 0% 3837Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 4927m 31% 11213Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 7662m 48% 11551Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 185m 1% 3064Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 90m 0% 1247Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 97m 0% 4953Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 65m 0% 14810Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 5548m 34% 14640Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 5295m 33% 14616Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 76m 0% 14777Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 5959m 37% 14645Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 2752m 17% 14817Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1368m 8% 2589Mi 4% 23:04:47 DEBUG --- stderr --- 23:04:47 DEBUG 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "ds-cts-1" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "ds-idrepo-1" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "0" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "ds-cts-1" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "ds-idrepo-1" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "0" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "214.6" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG ----- output ----- 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:30 DEBUG { 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG { 23:05:30 DEBUG "data": { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "result": [ 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG { 23:05:30 DEBUG "result": [ 23:05:30 DEBUG "metric": { 23:05:30 DEBUG { 23:05:30 DEBUG "pod": "ds-idrepo-2" 23:05:30 DEBUG "metric": { 23:05:30 DEBUG }, 23:05:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:05:30 DEBUG "value": [ 23:05:30 DEBUG }, 23:05:30 DEBUG 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "ds-idrepo-1" 23:05:30 DEBUG 23:05:30 DEBUG }, 23:05:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:05:30 DEBUG "value": [ 23:05:30 DEBUG }, 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG "0" 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG ] 23:05:30 DEBUG "0.004229780780965842" 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "0" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "0.2" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "instance": "10.142.0.17:9100" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "instance": "10.142.0.61:9100" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "7063580.6" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "ds-idrepo-2" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "ds-idrepo-1" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "0" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "instance": "10.142.0.17:9100" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "instance": "10.142.0.61:9100" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "0.0013333333333349628" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "ds-cts-1" 23:05:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:05:30 DEBUG }, 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "ds-idrepo-1" 23:05:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:05:30 DEBUG }, 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "0" 23:05:30 DEBUG "643.7" 23:05:30 DEBUG ] 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447530" 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG ----- output ----- 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:05:30 DEBUG { 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG { 23:05:30 DEBUG "data": { 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:30 DEBUG "data": { 23:05:30 DEBUG "result": [ 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG { 23:05:30 DEBUG { 23:05:30 DEBUG "result": [ 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG { 23:05:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:05:30 DEBUG "data": { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG }, 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:05:30 DEBUG "value": [ 23:05:30 DEBUG "result": [ 23:05:30 DEBUG }, 23:05:30 DEBUG 23:05:30 DEBUG { 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG "metric": { 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "instance": "10.142.0.17:9100" 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG }, 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "value": [ 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG }, 23:05:30 DEBUG [...] 23:05:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG }, 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG 23:05:30 DEBUG "value": [ 23:05:30 DEBUG "214.7" 23:05:30 DEBUG 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG ] 23:05:30 DEBUG "instance": "10.142.0.61:9100" 23:05:30 DEBUG "0" 23:05:30 DEBUG } 23:05:30 DEBUG }, 23:05:30 DEBUG ] 23:05:30 DEBUG ] 23:05:30 DEBUG "value": [ 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG "0.0003333333333333262" 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "643.7333333333333" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447530" 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:05:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:05:30 DEBUG }, 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:05:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:05:30 DEBUG }, 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "227.92689464541567" 23:05:30 DEBUG "14593777664" 23:05:30 DEBUG ] 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "1.3052262393410097" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "pod": "ds-idrepo-2" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "pod": "ds-idrepo-1" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "0" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447530" 23:05:30 INFO [http_cmd]: http status code OK 23:05:30 DEBUG --- status code --- 23:05:30 DEBUG http status code is 200 (expected 200) 23:05:30 DEBUG --- http response --- 23:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:30 DEBUG ----- output ----- 23:05:30 DEBUG { 23:05:30 DEBUG "status": "success", 23:05:30 DEBUG "data": { 23:05:30 DEBUG "resultType": "vector", 23:05:30 DEBUG "result": [ 23:05:30 DEBUG { 23:05:30 DEBUG "metric": { 23:05:30 DEBUG "instance": "10.142.0.17:9100" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG [...] 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG 23:05:30 DEBUG "instance": "10.142.0.61:9100" 23:05:30 DEBUG }, 23:05:30 DEBUG "value": [ 23:05:30 DEBUG 1684447530, 23:05:30 DEBUG "7066729.699999999" 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG ] 23:05:30 DEBUG } 23:05:30 DEBUG } 23:05:43 INFO 23:05:43 INFO [loop_until]: kubectl --namespace=xlou top pods 23:05:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:05:43 INFO [loop_until]: OK (rc = 0) 23:05:43 DEBUG --- stdout --- 23:05:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 2783m 8969Mi am-7849cf7bdb-vcmhd 2100m 8972Mi am-7849cf7bdb-vnjrd 2170m 8963Mi ds-cts-0 3169m 13834Mi ds-cts-1 3396m 13840Mi ds-cts-2 3746m 13817Mi ds-idrepo-0 1697m 13822Mi ds-idrepo-1 13m 13826Mi ds-idrepo-2 15m 13822Mi end-user-ui-787cb4f6b4-j246m 1m 3Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 4m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 552m 64Mi 23:05:43 DEBUG --- stderr --- 23:05:43 DEBUG 23:05:47 INFO 23:05:47 INFO [loop_until]: kubectl --namespace=xlou top node 23:05:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:05:47 INFO [loop_until]: OK (rc = 0) 23:05:47 DEBUG --- stdout --- 23:05:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 2009m 12% 11154Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 3836Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 3283m 20% 11213Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 4699m 29% 11538Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 158m 0% 3065Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 87m 0% 1243Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 90m 0% 4954Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14812Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 2565m 16% 14639Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 3911m 24% 14629Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 76m 0% 14779Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 3009m 18% 14639Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 1195m 7% 14819Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 721m 4% 2092Mi 3% 23:05:47 DEBUG --- stderr --- 23:05:47 DEBUG 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "ds-cts-1" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "ds-idrepo-1" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "ds-cts-1" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "ds-idrepo-1" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "ds-idrepo-0" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0.1698136500706614" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:06:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:06:30 DEBUG }, 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:06:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:06:30 DEBUG }, 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0.2" 23:06:30 DEBUG "0" 23:06:30 DEBUG ] 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "ds-idrepo-2" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "ds-idrepo-1" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:06:30 DEBUG "instance": "10.142.0.17:9100" 23:06:30 DEBUG }, 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "ds-idrepo-0" 23:06:30 DEBUG "instance": "10.142.0.61:9100" 23:06:30 DEBUG }, 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0.5015770378236246" 23:06:30 DEBUG "3458.233333333333" 23:06:30 DEBUG ] 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447590" 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "instance": "10.142.0.17:9100" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "instance": "10.142.0.61:9100" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0.0007000000000005002" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "ds-idrepo-2" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "ds-idrepo-1" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "ds-cts-1" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "ds-idrepo-1" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG "metric": { 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG }, 23:06:30 DEBUG --- http response --- 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG "value": [ 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG 23:06:30 DEBUG { 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:30 DEBUG [...] 23:06:30 DEBUG "data": { 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG { 23:06:30 DEBUG 23:06:30 DEBUG "result": [ 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG 23:06:30 DEBUG { 23:06:30 DEBUG "data": { 23:06:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG }, 23:06:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:06:30 DEBUG "result": [ 23:06:30 DEBUG "value": [ 23:06:30 DEBUG }, 23:06:30 DEBUG { 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "0" 23:06:30 DEBUG 23:06:30 DEBUG "instance": "10.142.0.17:9100" 23:06:30 DEBUG ] 23:06:30 DEBUG 23:06:30 DEBUG }, 23:06:30 DEBUG } 23:06:30 DEBUG 23:06:30 DEBUG "value": [ 23:06:30 DEBUG ] 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG } 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG } 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG "pod": "ds-idrepo-0" 23:06:30 DEBUG 23:06:30 DEBUG }, 23:06:30 DEBUG 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "instance": "10.142.0.61:9100" 23:06:30 DEBUG "0" 23:06:30 DEBUG }, 23:06:30 DEBUG ] 23:06:30 DEBUG "value": [ 23:06:30 DEBUG } 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG ] 23:06:30 DEBUG "0.0006666666666668892" 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "ds-idrepo-0" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "396798749.4110033" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "14591967232" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "pod": "ds-idrepo-2" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "pod": "ds-idrepo-1" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447590" 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "instance": "10.142.0.17:9100" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447590" 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "instance": "10.142.0.61:9100" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "6271.299999999999" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:30 INFO [http_cmd]: http status code OK 23:06:30 DEBUG --- status code --- 23:06:30 DEBUG http status code is 200 (expected 200) 23:06:30 DEBUG --- http response --- 23:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:30 DEBUG ----- output ----- 23:06:30 DEBUG { 23:06:30 DEBUG "status": "success", 23:06:30 DEBUG "data": { 23:06:30 DEBUG "resultType": "vector", 23:06:30 DEBUG "result": [ 23:06:30 DEBUG { 23:06:30 DEBUG "metric": { 23:06:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG [...] 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG 23:06:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:06:30 DEBUG }, 23:06:30 DEBUG "value": [ 23:06:30 DEBUG 1684447590, 23:06:30 DEBUG "0.02617214764363314" 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG ] 23:06:30 DEBUG } 23:06:30 DEBUG } 23:06:43 INFO 23:06:43 INFO [loop_until]: kubectl --namespace=xlou top pods 23:06:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:06:43 INFO [loop_until]: OK (rc = 0) 23:06:43 DEBUG --- stdout --- 23:06:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 4Mi am-7849cf7bdb-2vjw5 303m 8962Mi am-7849cf7bdb-vcmhd 325m 8969Mi am-7849cf7bdb-vnjrd 301m 8964Mi ds-cts-0 723m 13839Mi ds-cts-1 1184m 13829Mi ds-cts-2 718m 13822Mi ds-idrepo-0 282m 13800Mi ds-idrepo-1 13m 13827Mi ds-idrepo-2 15m 13822Mi end-user-ui-787cb4f6b4-j246m 1m 5Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 6m 3347Mi lodemon-58f494d89b-c4l9j 4m 68Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 1m 64Mi 23:06:43 DEBUG --- stderr --- 23:06:43 DEBUG 23:06:47 INFO 23:06:47 INFO [loop_until]: kubectl --namespace=xlou top node 23:06:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:06:47 INFO [loop_until]: OK (rc = 0) 23:06:47 DEBUG --- stdout --- 23:06:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 417m 2% 11149Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 86m 0% 3831Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 450m 2% 11212Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 412m 2% 11539Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 161m 1% 3065Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 100m 0% 1244Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 96m 0% 4950Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14812Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 929m 5% 14587Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 792m 4% 14629Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 70m 0% 14781Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 1157m 7% 14642Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 418m 2% 14795Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 70m 0% 2091Mi 3% 23:06:47 DEBUG --- stderr --- 23:06:47 DEBUG 127.0.0.1 - - [18/May/2023 23:07:06] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:07:13] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:07:20] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:07:27] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "ds-cts-1" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "ds-idrepo-1" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "ds-cts-1" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "ds-idrepo-1" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0.005291102084483118" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0.16417833333333334" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0.2" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447650" 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "ds-idrepo-2" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "ds-idrepo-1" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "instance": "10.142.0.17:9100" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "instance": "10.142.0.61:9100" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "6130.433333333333" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0.03333333333333333" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "ds-idrepo-2" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "ds-idrepo-1" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "instance": "10.142.0.17:9100" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "instance": "10.142.0.61:9100" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0.0013333333333340155" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "ds-cts-1" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "ds-idrepo-1" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG } 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG } 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "instance": "10.142.0.17:9100" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "instance": "10.142.0.61:9100" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0.0003333333333333262" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0.03333333333333333" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:07:30 DEBUG "instance": "10.142.0.17:9100" 23:07:30 DEBUG }, 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:07:30 DEBUG "instance": "10.142.0.61:9100" 23:07:30 DEBUG }, 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "4947.542794036444" 23:07:30 DEBUG "17419" 23:07:30 DEBUG ] 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "14533173248" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "pod": "ds-idrepo-2" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "pod": "ds-idrepo-1" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 23:07:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447650" 23:07:30 INFO [http_cmd]: http status code OK 23:07:30 DEBUG --- status code --- 23:07:30 DEBUG http status code is 200 (expected 200) 23:07:30 DEBUG --- http response --- 23:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:30 DEBUG ----- output ----- 23:07:30 DEBUG { 23:07:30 DEBUG "status": "success", 23:07:30 DEBUG "data": { 23:07:30 DEBUG "resultType": "vector", 23:07:30 DEBUG "result": [ 23:07:30 DEBUG { 23:07:30 DEBUG "metric": { 23:07:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG [...] 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG 23:07:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:07:30 DEBUG }, 23:07:30 DEBUG "value": [ 23:07:30 DEBUG 1684447650, 23:07:30 DEBUG "0.5303982732234439" 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG ] 23:07:30 DEBUG } 23:07:30 DEBUG } 127.0.0.1 - - [18/May/2023 23:07:33] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:07:40] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:07:41] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 23:07:43 INFO 23:07:43 INFO [loop_until]: kubectl --namespace=xlou top pods 23:07:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:07:43 INFO [loop_until]: OK (rc = 0) 23:07:43 DEBUG --- stdout --- 23:07:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 5Mi am-7849cf7bdb-2vjw5 324m 8955Mi am-7849cf7bdb-vcmhd 322m 8969Mi am-7849cf7bdb-vnjrd 306m 8958Mi ds-cts-0 1085m 13788Mi ds-cts-1 915m 13787Mi ds-cts-2 937m 13767Mi ds-idrepo-0 456m 13792Mi ds-idrepo-1 139m 13773Mi ds-idrepo-2 161m 13771Mi end-user-ui-787cb4f6b4-j246m 1m 5Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 7m 3347Mi lodemon-58f494d89b-c4l9j 17m 71Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 580m 316Mi 23:07:43 DEBUG --- stderr --- 23:07:43 DEBUG 23:07:47 INFO 23:07:47 INFO [loop_until]: kubectl --namespace=xlou top node 23:07:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:07:47 INFO [loop_until]: OK (rc = 0) 23:07:47 DEBUG --- stdout --- 23:07:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 410m 2% 11143Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 71m 0% 3835Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 436m 2% 11210Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 401m 2% 11532Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 153m 0% 3066Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 107m 0% 1258Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 94m 0% 4955Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 181m 1% 14755Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 1129m 7% 14596Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 991m 6% 14578Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 207m 1% 14731Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 937m 5% 14596Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 402m 2% 14788Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 624m 3% 2139Mi 3% 23:07:47 DEBUG --- stderr --- 23:07:47 DEBUG 127.0.0.1 - - [18/May/2023 23:07:47] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:07:48] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:07:55] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:07:56] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:02] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:03] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:10] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:11] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:17] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:18] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:19] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:26] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:26] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:27] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "ds-cts-1" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "ds-idrepo-1" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "ds-cts-1" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "ds-idrepo-1" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "1.1046541458311943" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0.5956505106011866" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0.2" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "ds-idrepo-2" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "ds-idrepo-1" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "instance": "10.142.0.17:9100" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "instance": "10.142.0.61:9100" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "4825.9" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG { 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG "data": { 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:30 DEBUG "result": [ 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "pod": "ds-idrepo-2" 23:08:30 DEBUG "data": { 23:08:30 DEBUG }, 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "value": [ 23:08:30 DEBUG "result": [ 23:08:30 DEBUG 23:08:30 DEBUG { 23:08:30 DEBUG 23:08:30 DEBUG "metric": { 23:08:30 DEBUG 23:08:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:08:30 DEBUG [...] 23:08:30 DEBUG }, 23:08:30 DEBUG 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "ds-idrepo-1" 23:08:30 DEBUG 23:08:30 DEBUG }, 23:08:30 DEBUG [...] 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG 23:08:30 DEBUG "0" 23:08:30 DEBUG 23:08:30 DEBUG ] 23:08:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:08:30 DEBUG } 23:08:30 DEBUG }, 23:08:30 DEBUG ] 23:08:30 DEBUG "value": [ 23:08:30 DEBUG } 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG } 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "instance": "10.142.0.17:9100" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "instance": "10.142.0.61:9100" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0.0018333333333326132" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "ds-cts-1" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "ds-idrepo-1" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "instance": "10.142.0.17:9100" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "instance": "10.142.0.61:9100" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0.00033333333333350387" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "instance": "10.142.0.17:9100" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "instance": "10.142.0.61:9100" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "16700.266666666666" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "ds-idrepo-2" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "ds-idrepo-1" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "64968559.98376647" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "14543192064" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 23:08:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447710" 23:08:30 INFO [http_cmd]: http status code OK 23:08:30 DEBUG --- status code --- 23:08:30 DEBUG http status code is 200 (expected 200) 23:08:30 DEBUG --- http response --- 23:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:30 DEBUG ----- output ----- 23:08:30 DEBUG { 23:08:30 DEBUG "status": "success", 23:08:30 DEBUG "data": { 23:08:30 DEBUG "resultType": "vector", 23:08:30 DEBUG "result": [ 23:08:30 DEBUG { 23:08:30 DEBUG "metric": { 23:08:30 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG [...] 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG 23:08:30 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:08:30 DEBUG }, 23:08:30 DEBUG "value": [ 23:08:30 DEBUG 1684447710, 23:08:30 DEBUG "0.6697375680362925" 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG ] 23:08:30 DEBUG } 23:08:30 DEBUG } 127.0.0.1 - - [18/May/2023 23:08:34] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:35] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:36] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:42] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:43] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 23:08:43 INFO 23:08:43 INFO [loop_until]: kubectl --namespace=xlou top pods 23:08:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:43 INFO [loop_until]: OK (rc = 0) 23:08:43 DEBUG --- stdout --- 23:08:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 5Mi am-7849cf7bdb-2vjw5 315m 8954Mi am-7849cf7bdb-vcmhd 347m 8973Mi am-7849cf7bdb-vnjrd 306m 8959Mi ds-cts-0 705m 13776Mi ds-cts-1 1038m 13792Mi ds-cts-2 914m 13797Mi ds-idrepo-0 388m 13793Mi ds-idrepo-1 13m 13773Mi ds-idrepo-2 19m 13772Mi end-user-ui-787cb4f6b4-j246m 1m 5Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 38m 71Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 825m 198Mi 23:08:43 DEBUG --- stderr --- 23:08:43 DEBUG 127.0.0.1 - - [18/May/2023 23:08:44] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 23:08:47 INFO 23:08:47 INFO [loop_until]: kubectl --namespace=xlou top node 23:08:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:47 INFO [loop_until]: OK (rc = 0) 23:08:47 DEBUG --- stdout --- 23:08:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 417m 2% 11145Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 77m 0% 3836Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 439m 2% 11211Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 397m 2% 11533Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 162m 1% 3076Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 130m 0% 1260Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 90m 0% 4954Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 67m 0% 14757Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 855m 5% 14584Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 882m 5% 14605Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 76m 0% 14728Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 1143m 7% 14609Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 457m 2% 14789Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1021m 6% 2221Mi 3% 23:08:47 DEBUG --- stderr --- 23:08:47 DEBUG 127.0.0.1 - - [18/May/2023 23:08:51] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:51] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:52] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:08:59] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:00] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:01] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:01] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:08] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:09] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:10] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:10] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:17] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:18] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:19] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:20] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:26] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:27] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:28] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:29] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "ds-cts-1" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "ds-idrepo-1" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "ds-cts-1" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "ds-idrepo-1" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0.2" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447770" 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "instance": "10.142.0.17:9100" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "instance": "10.142.0.61:9100" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "7702.700000000001" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0.3741854583742196" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "ds-idrepo-2" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "ds-idrepo-1" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "ds-idrepo-2" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "ds-idrepo-1" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447770" 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "instance": "10.142.0.17:9100" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "instance": "10.142.0.61:9100" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0.0014333333333335454" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "ds-cts-1" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "ds-idrepo-1" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447770" 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "instance": "10.142.0.17:9100" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "instance": "10.142.0.61:9100" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "18515.833333333332" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "instance": "10.142.0.17:9100" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "instance": "10.142.0.61:9100" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "ds-idrepo-2" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "ds-idrepo-1" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447770" 23:09:30 INFO [http_cmd]: http status code OK 23:09:30 DEBUG --- status code --- 23:09:30 DEBUG http status code is 200 (expected 200) 23:09:30 DEBUG --- http response --- 23:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:30 DEBUG ----- output ----- 23:09:30 DEBUG { 23:09:30 DEBUG "status": "success", 23:09:30 DEBUG "data": { 23:09:30 DEBUG "resultType": "vector", 23:09:30 DEBUG "result": [ 23:09:30 DEBUG { 23:09:30 DEBUG "metric": { 23:09:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 23:09:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447770" 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG [...] 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG 23:09:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:09:30 DEBUG }, 23:09:30 DEBUG "value": [ 23:09:30 DEBUG 1684447770, 23:09:30 DEBUG "0" 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG ] 23:09:30 DEBUG } 23:09:30 DEBUG } 23:09:31 INFO [http_cmd]: http status code OK 23:09:31 DEBUG --- status code --- 23:09:31 DEBUG http status code is 200 (expected 200) 23:09:31 DEBUG --- http response --- 23:09:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:31 DEBUG ----- output ----- 23:09:31 DEBUG { 23:09:31 DEBUG "status": "success", 23:09:31 DEBUG "data": { 23:09:31 DEBUG "resultType": "vector", 23:09:31 DEBUG "result": [ 23:09:31 DEBUG { 23:09:31 DEBUG "metric": { 23:09:31 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:09:31 DEBUG }, 23:09:31 DEBUG "value": [ 23:09:31 DEBUG 23:09:31 DEBUG 23:09:31 DEBUG 23:09:31 DEBUG [...] 23:09:31 DEBUG 23:09:31 DEBUG 23:09:31 DEBUG 23:09:31 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:09:31 DEBUG }, 23:09:31 DEBUG "value": [ 23:09:31 DEBUG 1684447770, 23:09:31 DEBUG "14550339584" 23:09:31 DEBUG ] 23:09:31 DEBUG } 23:09:31 DEBUG ] 23:09:31 DEBUG } 23:09:31 DEBUG } 23:09:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447771" 23:09:31 INFO [http_cmd]: http status code OK 23:09:31 DEBUG --- status code --- 23:09:31 DEBUG http status code is 200 (expected 200) 23:09:31 DEBUG --- http response --- 23:09:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:31 DEBUG ----- output ----- 23:09:31 DEBUG { 23:09:31 DEBUG "status": "success", 23:09:31 DEBUG "data": { 23:09:31 DEBUG "resultType": "vector", 23:09:31 DEBUG "result": [ 23:09:31 DEBUG { 23:09:31 DEBUG "metric": { 23:09:31 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:09:31 DEBUG }, 23:09:31 DEBUG "value": [ 23:09:31 DEBUG 23:09:31 DEBUG 23:09:31 DEBUG 23:09:31 DEBUG [...] 23:09:31 DEBUG 23:09:31 DEBUG 23:09:31 DEBUG 23:09:31 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:09:31 DEBUG }, 23:09:31 DEBUG "value": [ 23:09:31 DEBUG 1684447771, 23:09:31 DEBUG "0.7085935476979731" 23:09:31 DEBUG ] 23:09:31 DEBUG } 23:09:31 DEBUG ] 23:09:31 DEBUG } 23:09:31 DEBUG } 127.0.0.1 - - [18/May/2023 23:09:35] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:36] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:37] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:38] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 23:09:43 INFO 23:09:43 INFO [loop_until]: kubectl --namespace=xlou top pods 23:09:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:43 INFO [loop_until]: OK (rc = 0) 23:09:43 DEBUG --- stdout --- 23:09:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 5Mi am-7849cf7bdb-2vjw5 301m 8954Mi am-7849cf7bdb-vcmhd 346m 8967Mi am-7849cf7bdb-vnjrd 298m 8958Mi ds-cts-0 1050m 13776Mi ds-cts-1 992m 13805Mi ds-cts-2 733m 13784Mi ds-idrepo-0 237m 13793Mi ds-idrepo-1 13m 13773Mi ds-idrepo-2 18m 13771Mi end-user-ui-787cb4f6b4-j246m 1m 5Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 42m 71Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 812m 305Mi 23:09:43 DEBUG --- stderr --- 23:09:43 DEBUG 127.0.0.1 - - [18/May/2023 23:09:45] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:46] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:46] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 23:09:47 INFO 23:09:47 INFO [loop_until]: kubectl --namespace=xlou top node 23:09:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:47 INFO [loop_until]: OK (rc = 0) 127.0.0.1 - - [18/May/2023 23:09:47] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 23:09:47 DEBUG --- stdout --- 23:09:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 391m 2% 11141Mi 18% gke-xlou-cdm-default-pool-cab466d1-8xtg 81m 0% 3836Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 444m 2% 11208Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 405m 2% 11529Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 159m 1% 3042Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 143m 0% 1260Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 89m 0% 4954Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 68m 0% 14756Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 1137m 7% 14594Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 793m 4% 14593Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 77m 0% 14730Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 1015m 6% 14619Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 290m 1% 14790Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 888m 5% 2333Mi 3% 23:09:47 DEBUG --- stderr --- 23:09:47 DEBUG 127.0.0.1 - - [18/May/2023 23:09:54] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:55] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:55] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:56] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:09:57] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:04] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:05] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:06] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:06] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:07] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:14] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:15] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:15] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:16] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:17] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:24] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:25] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:25] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:26] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:27] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "ds-cts-1" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "ds-idrepo-1" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "ds-cts-1" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "ds-idrepo-1" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0.3611146653145634" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "2.1041556193290187" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0.2" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447830" 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG { 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG "data": { 23:10:30 DEBUG { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG "data": { 23:10:30 DEBUG { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "result": [ 23:10:30 DEBUG "pod": "ds-idrepo-2" 23:10:30 DEBUG { 23:10:30 DEBUG }, 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "value": [ 23:10:30 DEBUG "instance": "10.142.0.17:9100" 23:10:30 DEBUG 23:10:30 DEBUG }, 23:10:30 DEBUG 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "ds-idrepo-1" 23:10:30 DEBUG 23:10:30 DEBUG }, 23:10:30 DEBUG 23:10:30 DEBUG "value": [ 23:10:30 DEBUG "instance": "10.142.0.61:9100" 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG }, 23:10:30 DEBUG "0" 23:10:30 DEBUG "value": [ 23:10:30 DEBUG ] 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG } 23:10:30 DEBUG "5806.266666666666" 23:10:30 DEBUG ] 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "ds-idrepo-2" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "ds-idrepo-1" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "instance": "10.142.0.17:9100" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "instance": "10.142.0.61:9100" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0.0012000000000000454" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG { 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:30 DEBUG "data": { 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG { 23:10:30 DEBUG "result": [ 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG { 23:10:30 DEBUG "data": { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "pod": "ds-cts-1" 23:10:30 DEBUG "result": [ 23:10:30 DEBUG }, 23:10:30 DEBUG { 23:10:30 DEBUG "value": [ 23:10:30 DEBUG "metric": { 23:10:30 DEBUG 23:10:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:10:30 DEBUG 23:10:30 DEBUG }, 23:10:30 DEBUG 23:10:30 DEBUG "value": [ 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG "pod": "ds-idrepo-1" 23:10:30 DEBUG 23:10:30 DEBUG }, 23:10:30 DEBUG 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:10:30 DEBUG "0" 23:10:30 DEBUG }, 23:10:30 DEBUG ] 23:10:30 DEBUG "value": [ 23:10:30 DEBUG } 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG ] 23:10:30 DEBUG "0" 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447830" 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "instance": "10.142.0.17:9100" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "instance": "10.142.0.61:9100" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "18033.1" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447830" 23:10:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447830" 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "instance": "10.142.0.17:9100" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "instance": "10.142.0.61:9100" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0.0006666666666668892" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:30 INFO [http_cmd]: http status code OK 23:10:30 DEBUG --- status code --- 23:10:30 DEBUG http status code is 200 (expected 200) 23:10:30 DEBUG --- http response --- 23:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:30 DEBUG ----- output ----- 23:10:30 DEBUG { 23:10:30 DEBUG "status": "success", 23:10:30 DEBUG "data": { 23:10:30 DEBUG "resultType": "vector", 23:10:30 DEBUG "result": [ 23:10:30 DEBUG { 23:10:30 DEBUG "metric": { 23:10:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG [...] 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG 23:10:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:10:30 DEBUG }, 23:10:30 DEBUG "value": [ 23:10:30 DEBUG 1684447830, 23:10:30 DEBUG "0" 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG ] 23:10:30 DEBUG } 23:10:30 DEBUG } 23:10:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447831" 23:10:31 INFO [http_cmd]: http status code OK 23:10:31 DEBUG --- status code --- 23:10:31 DEBUG http status code is 200 (expected 200) 23:10:31 DEBUG --- http response --- 23:10:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:31 DEBUG ----- output ----- 23:10:31 DEBUG { 23:10:31 DEBUG "status": "success", 23:10:31 DEBUG "data": { 23:10:31 DEBUG "resultType": "vector", 23:10:31 DEBUG "result": [ 23:10:31 DEBUG { 23:10:31 DEBUG "metric": { 23:10:31 DEBUG "pod": "ds-idrepo-2" 23:10:31 DEBUG }, 23:10:31 DEBUG "value": [ 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG [...] 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG "pod": "ds-idrepo-1" 23:10:31 DEBUG }, 23:10:31 DEBUG "value": [ 23:10:31 DEBUG 1684447831, 23:10:31 DEBUG "0" 23:10:31 DEBUG ] 23:10:31 DEBUG } 23:10:31 DEBUG ] 23:10:31 DEBUG } 23:10:31 DEBUG } 23:10:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447831" 23:10:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447831" 23:10:31 INFO [http_cmd]: http status code OK 23:10:31 DEBUG --- status code --- 23:10:31 DEBUG http status code is 200 (expected 200) 23:10:31 DEBUG --- http response --- 23:10:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:10:31 DEBUG ----- output ----- 23:10:31 DEBUG { 23:10:31 DEBUG "status": "success", 23:10:31 DEBUG "data": { 23:10:31 DEBUG "resultType": "vector", 23:10:31 DEBUG "result": [ 23:10:31 DEBUG { 23:10:31 DEBUG "metric": { 23:10:31 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:10:31 DEBUG }, 23:10:31 DEBUG "value": [ 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG [...] 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG "pod": "idm-78f4b47cb9-bksrb" 23:10:31 DEBUG }, 23:10:31 DEBUG "value": [ 23:10:31 DEBUG 1684447831, 23:10:31 DEBUG "0" 23:10:31 DEBUG ] 23:10:31 DEBUG } 23:10:31 DEBUG ] 23:10:31 DEBUG } 23:10:31 DEBUG } 23:10:31 INFO [http_cmd]: http status code OK 23:10:31 DEBUG --- status code --- 23:10:31 DEBUG http status code is 200 (expected 200) 23:10:31 DEBUG --- http response --- 23:10:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:31 DEBUG ----- output ----- 23:10:31 DEBUG { 23:10:31 DEBUG "status": "success", 23:10:31 DEBUG "data": { 23:10:31 DEBUG "resultType": "vector", 23:10:31 DEBUG "result": [ 23:10:31 DEBUG { 23:10:31 DEBUG "metric": { 23:10:31 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:10:31 DEBUG }, 23:10:31 DEBUG "value": [ 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG [...] 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:10:31 DEBUG }, 23:10:31 DEBUG "value": [ 23:10:31 DEBUG 1684447831, 23:10:31 DEBUG "14560817152" 23:10:31 DEBUG ] 23:10:31 DEBUG } 23:10:31 DEBUG ] 23:10:31 DEBUG } 23:10:31 DEBUG } 23:10:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447831" 23:10:31 INFO [http_cmd]: http status code OK 23:10:31 DEBUG --- status code --- 23:10:31 DEBUG http status code is 200 (expected 200) 23:10:31 DEBUG --- http response --- 23:10:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:31 DEBUG ----- output ----- 23:10:31 DEBUG { 23:10:31 DEBUG "status": "success", 23:10:31 DEBUG "data": { 23:10:31 DEBUG "resultType": "vector", 23:10:31 DEBUG "result": [ 23:10:31 DEBUG { 23:10:31 DEBUG "metric": { 23:10:31 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:10:31 DEBUG }, 23:10:31 DEBUG "value": [ 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG [...] 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG 23:10:31 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:10:31 DEBUG }, 23:10:31 DEBUG "value": [ 23:10:31 DEBUG 1684447831, 23:10:31 DEBUG "0.34631765764845807" 23:10:31 DEBUG ] 23:10:31 DEBUG } 23:10:31 DEBUG ] 23:10:31 DEBUG } 23:10:31 DEBUG } 127.0.0.1 - - [18/May/2023 23:10:34] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:35] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:36] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:37] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:37] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 23:10:43 INFO 23:10:43 INFO [loop_until]: kubectl --namespace=xlou top pods 23:10:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:43 INFO [loop_until]: OK (rc = 0) 23:10:43 DEBUG --- stdout --- 23:10:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 5Mi am-7849cf7bdb-2vjw5 313m 8954Mi am-7849cf7bdb-vcmhd 323m 8967Mi am-7849cf7bdb-vnjrd 301m 8958Mi ds-cts-0 806m 13784Mi ds-cts-1 1035m 13811Mi ds-cts-2 716m 13788Mi ds-idrepo-0 240m 13792Mi ds-idrepo-1 13m 13774Mi ds-idrepo-2 17m 13771Mi end-user-ui-787cb4f6b4-j246m 1m 5Mi idm-78f4b47cb9-bksrb 6m 2478Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 56m 71Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 669m 412Mi 23:10:43 DEBUG --- stderr --- 23:10:43 DEBUG 127.0.0.1 - - [18/May/2023 23:10:44] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:45] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:46] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:47] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 23:10:47 INFO 23:10:47 INFO [loop_until]: kubectl --namespace=xlou top node 23:10:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:47 INFO [loop_until]: OK (rc = 0) 23:10:47 DEBUG --- stdout --- 23:10:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 392m 2% 11142Mi 18% gke-xlou-cdm-default-pool-cab466d1-8xtg 78m 0% 3834Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 432m 2% 11209Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 395m 2% 11530Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 156m 0% 3044Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 155m 0% 1256Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 88m 0% 4953Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 65m 0% 14759Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 829m 5% 14605Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 794m 4% 14597Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 70m 0% 14730Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 1105m 6% 14631Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 287m 1% 14790Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 661m 4% 2438Mi 4% 23:10:47 DEBUG --- stderr --- 23:10:47 DEBUG 127.0.0.1 - - [18/May/2023 23:10:47] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:54] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:55] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:56] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:57] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:57] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:10:58] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:05] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:06] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:07] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:07] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:08] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:09] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:16] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:16] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:17] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:18] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:19] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:20] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:26] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:27] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:28] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:29] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:30] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "pod": "ds-cts-1" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "ds-idrepo-1" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0" 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "pod": "ds-cts-1" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "ds-idrepo-1" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0" 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0" 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447890" 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:11:30 DEBUG "instance": "10.142.0.17:9100" 23:11:30 DEBUG }, 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:11:30 DEBUG "instance": "10.142.0.61:9100" 23:11:30 DEBUG }, 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0" 23:11:30 DEBUG "5076.633333333333" 23:11:30 DEBUG ] 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "ds-cts-2" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0" 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0.2" 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447890" 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "pod": "ds-idrepo-2" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "ds-idrepo-1" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0" 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG ] 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG } 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG ] 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG } 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:30 DEBUG } 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "pod": "ds-idrepo-2" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "ds-idrepo-1" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0" 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "instance": "10.142.0.17:9100" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "instance": "10.142.0.61:9100" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0.0015333333333330756" 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG { 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG "data": { 23:11:30 DEBUG { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG "data": { 23:11:30 DEBUG { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "result": [ 23:11:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:11:30 DEBUG { 23:11:30 DEBUG }, 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "value": [ 23:11:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:11:30 DEBUG 23:11:30 DEBUG }, 23:11:30 DEBUG 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:11:30 DEBUG 23:11:30 DEBUG }, 23:11:30 DEBUG 23:11:30 DEBUG "value": [ 23:11:30 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG }, 23:11:30 DEBUG "0" 23:11:30 DEBUG "value": [ 23:11:30 DEBUG ] 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG } 23:11:30 DEBUG "0" 23:11:30 DEBUG ] 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447890" 23:11:30 INFO [http_cmd]: http status code OK 23:11:30 DEBUG --- status code --- 23:11:30 DEBUG http status code is 200 (expected 200) 23:11:30 DEBUG --- http response --- 23:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:30 DEBUG ----- output ----- 23:11:30 DEBUG { 23:11:30 DEBUG "status": "success", 23:11:30 DEBUG "data": { 23:11:30 DEBUG "resultType": "vector", 23:11:30 DEBUG "result": [ 23:11:30 DEBUG { 23:11:30 DEBUG "metric": { 23:11:30 DEBUG "pod": "ds-cts-1" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG [...] 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG 23:11:30 DEBUG "pod": "ds-idrepo-1" 23:11:30 DEBUG }, 23:11:30 DEBUG "value": [ 23:11:30 DEBUG 1684447890, 23:11:30 DEBUG "0" 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG ] 23:11:30 DEBUG } 23:11:30 DEBUG } 23:11:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447891" 23:11:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447891" 23:11:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447891" 23:11:31 INFO [http_cmd]: http status code OK 23:11:31 DEBUG --- status code --- 23:11:31 INFO [http_cmd]: http status code OK 23:11:31 DEBUG http status code is 200 (expected 200) 23:11:31 DEBUG --- status code --- 23:11:31 DEBUG --- http response --- 23:11:31 DEBUG http status code is 200 (expected 200) 23:11:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:31 DEBUG --- http response --- 23:11:31 DEBUG ----- output ----- 23:11:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:31 DEBUG { 23:11:31 DEBUG ----- output ----- 23:11:31 DEBUG "status": "success", 23:11:31 DEBUG { 23:11:31 DEBUG "data": { 23:11:31 DEBUG "status": "success", 23:11:31 DEBUG "resultType": "vector", 23:11:31 DEBUG "data": { 23:11:31 DEBUG "result": [ 23:11:31 DEBUG "resultType": "vector", 23:11:31 DEBUG { 23:11:31 INFO [http_cmd]: http status code OK 23:11:31 DEBUG "result": [ 23:11:31 DEBUG "metric": { 23:11:31 DEBUG --- status code --- 23:11:31 DEBUG { 23:11:31 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:11:31 DEBUG http status code is 200 (expected 200) 23:11:31 DEBUG "metric": { 23:11:31 DEBUG }, 23:11:31 DEBUG --- http response --- 23:11:31 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:11:31 DEBUG "value": [ 23:11:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:31 DEBUG }, 23:11:31 DEBUG 23:11:31 DEBUG ----- output ----- 23:11:31 DEBUG "value": [ 23:11:31 DEBUG 23:11:31 DEBUG { 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG "status": "success", 23:11:31 DEBUG 23:11:31 DEBUG [...] 23:11:31 DEBUG "data": { 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG "resultType": "vector", 23:11:31 DEBUG [...] 23:11:31 DEBUG 23:11:31 DEBUG "result": [ 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG { 23:11:31 DEBUG 23:11:31 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:11:31 DEBUG "metric": { 23:11:31 DEBUG 23:11:31 DEBUG }, 23:11:31 DEBUG "instance": "10.142.0.17:9100" 23:11:31 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:11:31 DEBUG "value": [ 23:11:31 DEBUG }, 23:11:31 DEBUG }, 23:11:31 DEBUG 1684447891, 23:11:31 DEBUG "value": [ 23:11:31 DEBUG "value": [ 23:11:31 DEBUG "0" 23:11:31 DEBUG 23:11:31 DEBUG 1684447891, 23:11:31 DEBUG ] 23:11:31 DEBUG 23:11:31 DEBUG "0" 23:11:31 DEBUG } 23:11:31 DEBUG 23:11:31 DEBUG ] 23:11:31 DEBUG ] 23:11:31 DEBUG [...] 23:11:31 DEBUG } 23:11:31 DEBUG } 23:11:31 DEBUG 23:11:31 DEBUG ] 23:11:31 DEBUG } 23:11:31 DEBUG 23:11:31 DEBUG } 23:11:31 DEBUG 23:11:31 DEBUG } 23:11:31 DEBUG "instance": "10.142.0.61:9100" 23:11:31 DEBUG }, 23:11:31 DEBUG "value": [ 23:11:31 DEBUG 1684447891, 23:11:31 DEBUG "8334.033333333333" 23:11:31 DEBUG ] 23:11:31 DEBUG } 23:11:31 DEBUG ] 23:11:31 DEBUG } 23:11:31 DEBUG } 23:11:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447891" 23:11:31 INFO [http_cmd]: http status code OK 23:11:31 DEBUG --- status code --- 23:11:31 DEBUG http status code is 200 (expected 200) 23:11:31 DEBUG --- http response --- 23:11:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:31 DEBUG ----- output ----- 23:11:31 DEBUG { 23:11:31 DEBUG "status": "success", 23:11:31 DEBUG "data": { 23:11:31 DEBUG "resultType": "vector", 23:11:31 DEBUG "result": [ 23:11:31 DEBUG { 23:11:31 DEBUG "metric": { 23:11:31 DEBUG "instance": "10.142.0.17:9100" 23:11:31 DEBUG }, 23:11:31 DEBUG "value": [ 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG [...] 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG "instance": "10.142.0.61:9100" 23:11:31 DEBUG }, 23:11:31 DEBUG "value": [ 23:11:31 DEBUG 1684447891, 23:11:31 DEBUG "0.00033333333333350387" 23:11:31 DEBUG ] 23:11:31 DEBUG } 23:11:31 DEBUG ] 23:11:31 DEBUG } 23:11:31 DEBUG } 23:11:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447891" 23:11:31 INFO [http_cmd]: http status code OK 23:11:31 DEBUG --- status code --- 23:11:31 DEBUG http status code is 200 (expected 200) 23:11:31 DEBUG --- http response --- 23:11:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:31 DEBUG ----- output ----- 23:11:31 DEBUG { 23:11:31 DEBUG "status": "success", 23:11:31 DEBUG "data": { 23:11:31 DEBUG "resultType": "vector", 23:11:31 DEBUG "result": [ 23:11:31 DEBUG { 23:11:31 DEBUG "metric": { 23:11:31 DEBUG "pod": "ds-idrepo-2" 23:11:31 DEBUG }, 23:11:31 DEBUG "value": [ 23:11:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447891" 23:11:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447891" 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447891" 23:11:31 DEBUG [...] 127.0.0.1 - - [18/May/2023 23:11:31] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 23:11:31 INFO [http_cmd]: http status code OK 23:11:31 DEBUG 23:11:31 DEBUG --- status code --- 23:11:31 INFO [http_cmd]: http status code OK 23:11:31 DEBUG 23:11:31 DEBUG http status code is 200 (expected 200) 23:11:31 DEBUG --- status code --- 23:11:31 DEBUG 23:11:31 DEBUG --- http response --- 23:11:31 DEBUG http status code is 200 (expected 200) 23:11:31 DEBUG "pod": "ds-idrepo-1" 23:11:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:11:31 DEBUG --- http response --- 23:11:31 DEBUG }, 23:11:31 DEBUG ----- output ----- 23:11:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:31 DEBUG "value": [ 23:11:31 DEBUG { 23:11:31 INFO [http_cmd]: http status code OK 23:11:31 DEBUG ----- output ----- 23:11:31 DEBUG 1684447891, 23:11:31 DEBUG "status": "success", 23:11:31 DEBUG --- status code --- 23:11:31 DEBUG { 23:11:31 DEBUG "0" 23:11:31 DEBUG "data": { 23:11:31 DEBUG http status code is 200 (expected 200) 23:11:31 DEBUG "status": "success", 23:11:31 DEBUG ] 23:11:31 DEBUG "resultType": "vector", 23:11:31 DEBUG --- http response --- 23:11:31 DEBUG "data": { 23:11:31 DEBUG } 23:11:31 DEBUG "result": [ 23:11:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:31 DEBUG "resultType": "vector", 23:11:31 DEBUG ] 23:11:31 DEBUG { 23:11:31 DEBUG ----- output ----- 23:11:31 DEBUG "result": [ 23:11:31 DEBUG } 23:11:31 DEBUG "metric": { 23:11:31 DEBUG { 23:11:31 DEBUG { 23:11:31 DEBUG } 23:11:31 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:11:31 DEBUG "status": "success", 23:11:31 DEBUG "metric": { 23:11:31 DEBUG }, 23:11:31 DEBUG "data": { 23:11:31 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:11:31 DEBUG "value": [ 23:11:31 DEBUG "resultType": "vector", 23:11:31 DEBUG }, 23:11:31 DEBUG 23:11:31 DEBUG "result": [ 23:11:31 DEBUG "value": [ 23:11:31 DEBUG 23:11:31 DEBUG { 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG "metric": { 23:11:31 DEBUG 23:11:31 DEBUG [...] 23:11:31 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG }, 23:11:31 DEBUG [...] 23:11:31 DEBUG 23:11:31 DEBUG "value": [ 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG "pod": "overseer-0-67d979487f-8b45m" 23:11:31 DEBUG 23:11:31 DEBUG 23:11:31 DEBUG }, 23:11:31 DEBUG 23:11:31 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:11:31 DEBUG "value": [ 23:11:31 DEBUG [...] 23:11:31 DEBUG }, 23:11:31 DEBUG 1684447891, 23:11:31 DEBUG 23:11:31 DEBUG "value": [ 23:11:31 DEBUG "3934302.3620380806" 23:11:31 DEBUG 23:11:31 DEBUG 1684447891, 23:11:31 DEBUG ] 23:11:31 DEBUG 23:11:31 DEBUG "14575108096" 23:11:31 DEBUG } 23:11:31 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:11:31 DEBUG ] 23:11:31 DEBUG ] 23:11:31 DEBUG }, 23:11:31 DEBUG } 23:11:31 DEBUG } 23:11:31 DEBUG "value": [ 23:11:31 DEBUG ] 23:11:31 DEBUG } 23:11:31 DEBUG 1684447891, 23:11:31 DEBUG } 23:11:31 DEBUG "0.608979224085497" 23:11:31 DEBUG } 23:11:31 DEBUG ] 23:11:31 DEBUG } 23:11:31 DEBUG ] 23:11:31 DEBUG } 23:11:31 DEBUG } 127.0.0.1 - - [18/May/2023 23:11:37] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:38] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:39] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:40] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:41] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:41] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 23:11:44 INFO 23:11:44 INFO [loop_until]: kubectl --namespace=xlou top pods 23:11:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:44 INFO [loop_until]: OK (rc = 0) 23:11:44 DEBUG --- stdout --- 23:11:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 5Mi am-7849cf7bdb-2vjw5 314m 8954Mi am-7849cf7bdb-vcmhd 323m 8967Mi am-7849cf7bdb-vnjrd 304m 8958Mi ds-cts-0 827m 13796Mi ds-cts-1 865m 13826Mi ds-cts-2 801m 13831Mi ds-idrepo-0 587m 13793Mi ds-idrepo-1 13m 13774Mi ds-idrepo-2 17m 13771Mi end-user-ui-787cb4f6b4-j246m 1m 5Mi idm-78f4b47cb9-bksrb 6m 2479Mi idm-78f4b47cb9-gv5tc 5m 3347Mi lodemon-58f494d89b-c4l9j 63m 71Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 947m 572Mi 23:11:44 DEBUG --- stderr --- 23:11:44 DEBUG 23:11:47 INFO 23:11:47 INFO [loop_until]: kubectl --namespace=xlou top node 23:11:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:47 INFO [loop_until]: OK (rc = 0) 23:11:47 DEBUG --- stdout --- 23:11:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 406m 2% 11139Mi 18% gke-xlou-cdm-default-pool-cab466d1-8xtg 77m 0% 3836Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 435m 2% 11210Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 405m 2% 11525Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 151m 0% 3066Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 171m 1% 1256Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 90m 0% 4951Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14758Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 901m 5% 14611Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 876m 5% 14642Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 75m 0% 14732Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 941m 5% 14645Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 558m 3% 14790Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 948m 5% 2598Mi 4% 23:11:47 DEBUG --- stderr --- 23:11:47 DEBUG 127.0.0.1 - - [18/May/2023 23:11:48] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:49] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:50] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:51] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:51] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:52] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:11:59] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:00] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:01] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:01] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:02] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:03] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:04] "GET /monitoring/average?start_time=23-05-18_21:33:16&stop_time=23-05-18_21:49:39 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:11] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:11] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:12] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:13] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:14] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:15] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:16] "GET /monitoring/average?start_time=23-05-18_21:33:16&stop_time=23-05-18_21:49:39 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:22] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:23] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:24] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:25] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:26] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:26] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:27] "GET /monitoring/average?start_time=23-05-18_21:33:16&stop_time=23-05-18_21:49:39 HTTP/1.1" 200 - 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447950" 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447950" 23:12:30 INFO [http_cmd]: http status code OK 23:12:30 DEBUG --- status code --- 23:12:30 DEBUG http status code is 200 (expected 200) 23:12:30 DEBUG --- http response --- 23:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 23:12:30 DEBUG ----- output ----- 23:12:30 DEBUG { 23:12:30 DEBUG "status": "success", 23:12:30 DEBUG "data": { 23:12:30 DEBUG "resultType": "vector", 23:12:30 DEBUG "result": [ 23:12:30 DEBUG { 23:12:30 DEBUG "metric": { 23:12:30 DEBUG "pod": "ds-cts-1" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG [...] 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG "pod": "ds-idrepo-1" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 1684447950, 23:12:30 DEBUG "0" 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:30 INFO [http_cmd]: http status code OK 23:12:30 DEBUG --- status code --- 23:12:30 DEBUG http status code is 200 (expected 200) 23:12:30 DEBUG --- http response --- 23:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:30 DEBUG ----- output ----- 23:12:30 DEBUG { 23:12:30 DEBUG "status": "success", 23:12:30 DEBUG "data": { 23:12:30 DEBUG "resultType": "vector", 23:12:30 DEBUG "result": [ 23:12:30 DEBUG { 23:12:30 DEBUG "metric": { 23:12:30 DEBUG "pod": "ds-cts-1" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG [...] 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG "pod": "ds-idrepo-1" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 1684447950, 23:12:30 DEBUG "0" 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447950" 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_search_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447950" 23:12:30 INFO [http_cmd]: http status code OK 23:12:30 DEBUG --- status code --- 23:12:30 DEBUG http status code is 200 (expected 200) 23:12:30 DEBUG --- http response --- 23:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:30 DEBUG ----- output ----- 23:12:30 DEBUG { 23:12:30 DEBUG "status": "success", 23:12:30 DEBUG "data": { 23:12:30 DEBUG "resultType": "vector", 23:12:30 DEBUG "result": [ 23:12:30 DEBUG { 23:12:30 DEBUG "metric": { 23:12:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG [...] 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 1684447950, 23:12:30 DEBUG "0" 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:30 INFO [http_cmd]: http status code OK 23:12:30 DEBUG --- status code --- 23:12:30 DEBUG http status code is 200 (expected 200) 23:12:30 DEBUG --- http response --- 23:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:30 DEBUG ----- output ----- 23:12:30 DEBUG { 23:12:30 DEBUG "status": "success", 23:12:30 DEBUG "data": { 23:12:30 DEBUG "resultType": "vector", 23:12:30 DEBUG "result": [ 23:12:30 DEBUG { 23:12:30 DEBUG "metric": { 23:12:30 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG [...] 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 1684447950, 23:12:30 DEBUG "0.2" 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447950" 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447950" 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447950" 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447950" 23:12:30 INFO [http_cmd]: http status code OK 23:12:30 DEBUG --- status code --- 23:12:30 DEBUG http status code is 200 (expected 200) 23:12:30 DEBUG --- http response --- 23:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:12:30 DEBUG ----- output ----- 23:12:30 DEBUG { 23:12:30 DEBUG "status": "success", 23:12:30 DEBUG "data": { 23:12:30 DEBUG "resultType": "vector", 23:12:30 DEBUG "result": [ 23:12:30 DEBUG { 23:12:30 DEBUG "metric": { 23:12:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG [...] 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG "pod": "ds-idrepo-2" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 1684447950, 23:12:30 DEBUG "0.018030045153177306" 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:30 INFO [http_cmd]: http status code OK 23:12:30 DEBUG --- status code --- 23:12:30 DEBUG http status code is 200 (expected 200) 23:12:30 DEBUG --- http response --- 23:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:30 DEBUG ----- output ----- 23:12:30 DEBUG { 23:12:30 DEBUG "status": "success", 23:12:30 DEBUG "data": { 23:12:30 DEBUG "resultType": "vector", 23:12:30 DEBUG "result": [ 23:12:30 DEBUG { 23:12:30 DEBUG "metric": { 23:12:30 DEBUG "instance": "10.142.0.17:9100" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG [...] 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG "instance": "10.142.0.61:9100" 23:12:30 DEBUG }, 23:12:30 INFO [http_cmd]: http status code OK 23:12:30 DEBUG "value": [ 23:12:30 INFO [http_cmd]: http status code OK 23:12:30 DEBUG --- status code --- 23:12:30 DEBUG 1684447950, 23:12:30 DEBUG --- status code --- 23:12:30 DEBUG http status code is 200 (expected 200) 23:12:30 DEBUG "5745.566666666667" 23:12:30 DEBUG http status code is 200 (expected 200) 23:12:30 DEBUG --- http response --- 23:12:30 DEBUG ] 23:12:30 DEBUG --- http response --- 23:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:30 DEBUG } 23:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:12:30 DEBUG ----- output ----- 23:12:30 DEBUG ] 23:12:30 DEBUG ----- output ----- 23:12:30 DEBUG { 23:12:30 DEBUG } 23:12:30 DEBUG { 23:12:30 DEBUG "status": "success", 23:12:30 DEBUG } 23:12:30 DEBUG "status": "success", 23:12:30 DEBUG "data": { 23:12:30 DEBUG "data": { 23:12:30 DEBUG "resultType": "vector", 23:12:30 DEBUG "resultType": "vector", 23:12:30 DEBUG "result": [ 23:12:30 DEBUG "result": [ 23:12:30 DEBUG { 23:12:30 DEBUG { 23:12:30 DEBUG "metric": { 23:12:30 DEBUG "metric": { 23:12:30 DEBUG "pod": "ds-idrepo-2" 23:12:30 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:12:30 DEBUG }, 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG [...] 23:12:30 DEBUG [...] 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG "pod": "ds-idrepo-1" 23:12:30 DEBUG "pod": "ds-idrepo-2" 23:12:30 DEBUG }, 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 1684447950, 23:12:30 DEBUG 1684447950, 23:12:30 DEBUG "0" 23:12:30 DEBUG "0.7331133993135392" 23:12:30 DEBUG ] 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:30 DEBUG ] 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447950" 23:12:30 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447950" 23:12:30 INFO [http_cmd]: http status code OK 23:12:30 DEBUG --- status code --- 23:12:30 DEBUG http status code is 200 (expected 200) 23:12:30 DEBUG --- http response --- 23:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:30 DEBUG ----- output ----- 23:12:30 DEBUG { 23:12:30 DEBUG "status": "success", 23:12:30 DEBUG "data": { 23:12:30 DEBUG "resultType": "vector", 23:12:30 DEBUG "result": [ 23:12:30 DEBUG { 23:12:30 DEBUG "metric": { 23:12:30 DEBUG "pod": "ds-idrepo-2" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG [...] 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG 23:12:30 DEBUG "pod": "ds-idrepo-1" 23:12:30 DEBUG }, 23:12:30 DEBUG "value": [ 23:12:30 DEBUG 1684447950, 23:12:30 DEBUG "0" 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG ] 23:12:30 DEBUG } 23:12:30 DEBUG } 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG { 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG "data": { 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "result": [ 23:12:31 DEBUG { 23:12:31 DEBUG "metric": { 23:12:31 DEBUG "instance": "10.142.0.17:9100" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG "instance": "10.142.0.61:9100" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 1684447950, 23:12:31 DEBUG "0.0014666666666670381" 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447951" 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447951" 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG { 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG "data": { 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "result": [ 23:12:31 DEBUG { 23:12:31 DEBUG "metric": { 23:12:31 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG "0" 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447951" 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG { 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG "data": { 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "result": [ 23:12:31 DEBUG { 23:12:31 DEBUG "metric": { 23:12:31 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG "pod": "ds-idrepo-2" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG "0" 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG { 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG "data": { 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "result": [ 23:12:31 DEBUG { 23:12:31 DEBUG "metric": { 23:12:31 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG "0" 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447951" 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG { 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG "data": { 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "result": [ 23:12:31 DEBUG { 23:12:31 DEBUG "metric": { 23:12:31 DEBUG "pod": "ds-cts-1" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG "pod": "ds-idrepo-1" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG "0" 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447951" 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447951" 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG --- status code --- 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:31 DEBUG { 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG { 23:12:31 DEBUG "data": { 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "data": { 23:12:31 DEBUG "result": [ 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG { 23:12:31 DEBUG "result": [ 23:12:31 DEBUG "metric": { 23:12:31 DEBUG { 23:12:31 DEBUG "pod": "am-7849cf7bdb-2vjw5" 23:12:31 DEBUG "metric": { 23:12:31 DEBUG }, 23:12:31 DEBUG "instance": "10.142.0.17:9100" 23:12:31 DEBUG "value": [ 23:12:31 DEBUG }, 23:12:31 DEBUG 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG "pod": "am-7849cf7bdb-vcmhd" 23:12:31 DEBUG 23:12:31 DEBUG }, 23:12:31 DEBUG "instance": "10.142.0.61:9100" 23:12:31 DEBUG "value": [ 23:12:31 DEBUG }, 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG "0" 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG ] 23:12:31 DEBUG "18375.36666666667" 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1684447951" 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG { 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG "data": { 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "result": [ 23:12:31 DEBUG { 23:12:31 DEBUG "metric": { 23:12:31 DEBUG "instance": "10.142.0.17:9100" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG "instance": "10.142.0.61:9100" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG "0" 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447951" 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG { 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG "data": { 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "result": [ 23:12:31 DEBUG { 23:12:31 DEBUG "metric": { 23:12:31 DEBUG "pod": "ds-idrepo-2" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG "pod": "ds-idrepo-1" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG "0" 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1684447951" 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1684447951" 23:12:31 INFO [http_cmd]: curl --insecure -L -X GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1684447951" 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG --- http response --- 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG { 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:31 DEBUG "data": { 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG { 23:12:31 DEBUG "result": [ 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG { 23:12:31 DEBUG "data": { 23:12:31 DEBUG "metric": { 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "pod": "admin-ui-8666f85968-f8lws" 23:12:31 DEBUG "result": [ 23:12:31 DEBUG }, 23:12:31 DEBUG { 23:12:31 DEBUG "value": [ 23:12:31 DEBUG "metric": { 23:12:31 DEBUG 23:12:31 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:12:31 DEBUG 23:12:31 DEBUG }, 23:12:31 DEBUG 23:12:31 DEBUG "value": [ 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG "pod": "ds-idrepo-2" 23:12:31 DEBUG 23:12:31 DEBUG }, 23:12:31 DEBUG 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG "node": "gke-xlou-cdm-ds-c193de6c-gqp6" 23:12:31 INFO [http_cmd]: http status code OK 23:12:31 DEBUG "818.9543137058882" 23:12:31 DEBUG }, 23:12:31 DEBUG --- status code --- 23:12:31 DEBUG ] 23:12:31 DEBUG "value": [ 23:12:31 DEBUG http status code is 200 (expected 200) 23:12:31 DEBUG } 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG --- http response --- 23:12:31 DEBUG ] 23:12:31 DEBUG "14582460416" 23:12:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG ----- output ----- 23:12:31 DEBUG } 23:12:31 DEBUG } 23:12:31 DEBUG { 23:12:31 DEBUG ] 23:12:31 DEBUG "status": "success", 23:12:31 DEBUG } 23:12:31 DEBUG "data": { 23:12:31 DEBUG } 23:12:31 DEBUG "resultType": "vector", 23:12:31 DEBUG "result": [ 23:12:31 DEBUG { 23:12:31 DEBUG "metric": { 23:12:31 DEBUG "node": "gke-xlou-cdm-default-pool-cab466d1-h89r" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG [...] 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG 23:12:31 DEBUG "node": "gke-xlou-cdm-frontend-f357e02a-1hq2" 23:12:31 DEBUG }, 23:12:31 DEBUG "value": [ 23:12:31 DEBUG 1684447951, 23:12:31 DEBUG "0.3267498824213725" 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG ] 23:12:31 DEBUG } 23:12:31 DEBUG } 127.0.0.1 - - [18/May/2023 23:12:34] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:35] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:36] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:36] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:37] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:38] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:39] "GET /monitoring/average?start_time=23-05-18_21:33:16&stop_time=23-05-18_21:49:39 HTTP/1.1" 200 - 23:12:44 INFO 23:12:44 INFO [loop_until]: kubectl --namespace=xlou top pods 23:12:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:44 INFO [loop_until]: OK (rc = 0) 23:12:44 DEBUG --- stdout --- 23:12:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-8666f85968-f8lws 1m 5Mi am-7849cf7bdb-2vjw5 301m 8961Mi am-7849cf7bdb-vcmhd 330m 8967Mi am-7849cf7bdb-vnjrd 299m 8958Mi ds-cts-0 899m 13839Mi ds-cts-1 1084m 13836Mi ds-cts-2 880m 13787Mi ds-idrepo-0 242m 13794Mi ds-idrepo-1 11m 13773Mi ds-idrepo-2 18m 13771Mi end-user-ui-787cb4f6b4-j246m 1m 5Mi idm-78f4b47cb9-bksrb 6m 2479Mi idm-78f4b47cb9-gv5tc 7m 3347Mi lodemon-58f494d89b-c4l9j 62m 71Mi login-ui-57fddf97c8-n79g5 1m 3Mi overseer-0-67d979487f-8b45m 973m 1076Mi 23:12:44 DEBUG --- stderr --- 23:12:44 DEBUG 127.0.0.1 - - [18/May/2023 23:12:46] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:46] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:47] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 23:12:47 INFO 23:12:47 INFO [loop_until]: kubectl --namespace=xlou top node 23:12:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:48 INFO [loop_until]: OK (rc = 0) 23:12:48 DEBUG --- stdout --- 23:12:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-cab466d1-0gz0 395m 2% 11150Mi 19% gke-xlou-cdm-default-pool-cab466d1-8xtg 76m 0% 3833Mi 6% gke-xlou-cdm-default-pool-cab466d1-c9gd 450m 2% 11211Mi 19% gke-xlou-cdm-default-pool-cab466d1-d2r1 397m 2% 11530Mi 19% gke-xlou-cdm-default-pool-cab466d1-ggjd 151m 0% 3071Mi 5% gke-xlou-cdm-default-pool-cab466d1-h89r 159m 1% 1258Mi 2% gke-xlou-cdm-default-pool-cab466d1-xcr2 89m 0% 4955Mi 8% gke-xlou-cdm-ds-c193de6c-0rg3 66m 0% 14760Mi 25% gke-xlou-cdm-ds-c193de6c-3c57 942m 5% 14656Mi 24% gke-xlou-cdm-ds-c193de6c-45kq 935m 5% 14605Mi 24% gke-xlou-cdm-ds-c193de6c-7dtv 74m 0% 14730Mi 25% gke-xlou-cdm-ds-c193de6c-gqp6 1198m 7% 14654Mi 24% gke-xlou-cdm-ds-c193de6c-zp2c 294m 1% 14793Mi 25% gke-xlou-cdm-frontend-f357e02a-1hq2 1114m 7% 2758Mi 4% 23:12:48 DEBUG --- stderr --- 23:12:48 DEBUG 127.0.0.1 - - [18/May/2023 23:12:48] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:49] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:50] "GET /monitoring/average?start_time=23-05-18_21:16:59&stop_time=23-05-18_21:33:16 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:51] "GET /monitoring/average?start_time=23-05-18_21:33:16&stop_time=23-05-18_21:49:39 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:57] "GET /monitoring/average?start_time=23-05-18_19:55:05&stop_time=23-05-18_20:11:34 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:12:58] "GET /monitoring/average?start_time=23-05-18_20:11:34&stop_time=23-05-18_20:28:02 HTTP/1.1" 200 - 23:12:59 INFO Finished: True 23:12:59 INFO Waiting for threads to register finish flag 23:13:48 INFO Done. Have a nice day! :) 127.0.0.1 - - [18/May/2023 23:13:48] "GET /monitoring/stop HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:13:48] "GET /monitoring/average?start_time=23-05-18_20:28:02&stop_time=23-05-18_20:44:19 HTTP/1.1" 200 - 23:13:54 INFO File /tmp/lodemon_data-23-05-18_18:00:39/am_cts_reaper_cache_size.json does not exist. Skipping... 23:13:54 INFO File /tmp/lodemon_data-23-05-18_18:00:39/am_cts_reaper_search_seconds_total.json does not exist. Skipping... 23:13:54 INFO File /tmp/lodemon_data-23-05-18_18:00:39/ds_replication_replica_replayed_updates_conflicts_resolved.json does not exist. Skipping... 23:13:54 INFO File /tmp/lodemon_data-23-05-18_18:00:39/node_disk_read_bytes_total.json does not exist. Skipping... 23:13:54 INFO File /tmp/lodemon_data-23-05-18_18:00:39/node_disk_written_bytes_total.json does not exist. Skipping... 23:13:54 INFO File /tmp/lodemon_data-23-05-18_18:00:39/ds_backend_entry_count.json does not exist. Skipping... 127.0.0.1 - - [18/May/2023 23:14:02] "GET /monitoring/process HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:14:02] "GET /monitoring/average?start_time=23-05-18_20:44:19&stop_time=23-05-18_21:00:42 HTTP/1.1" 200 - 127.0.0.1 - - [18/May/2023 23:14:04] "GET /monitoring/average?start_time=23-05-18_21:00:42&stop_time=23-05-18_21:16:59 HTTP/1.1" 200 -